Transformers & Attention

In 2017, Google researchers published a paper titled "Attention Is All You Need". It introduced the Transformer architecture, which eliminated RNNs and became the foundation for every modern LLM (GPT, BERT, Claude, LLaMA).

[!TIP] Read the original paper — "Attention Is All You Need" (Vaswani et al., 2017). One of the most important papers in AI history.

1. The Problem with RNNs 🐢

• Sequential: Must process word 1, then word 2, then word 3. Slow! Cannot be parallelized on modern GPUs.
• Long-term Memory: Struggled with very long contexts — the gradient signal vanishes over many steps.
• Bottleneck: The entire sequence was compressed into a fixed-size context vector, losing information.

2. The Solution: Self-Attention 💡

Transformers process the entire sentence at once (parallelization). The core mechanism is Self-Attention — it allows every token to look at every other token to figure out context.

• Sentence: "The animal didn't cross the street because it was too tired."
• Attention: When the model processes "it", it pays high attention to "animal" and low attention to "street".

The Math of Self-Attention

Each token gets projected into three vectors:

• Query (Q): "What information am I looking for?"
• Key (K): "What information do I have?"
• Value (V): "What is the actual content I'll share?"

The attention formula:

Attention(Q, K, V) = softmax(QK^T / √d_k) × V

• QK^T — Dot product gives a score: how much does each query "match" each key?
• / √d_k — Scaling prevents the scores from getting too large (causes softmax to saturate)
• softmax(...) — Converts scores to probabilities (0 to 1, sum to 1)
• × V — Weight the value vectors by attention scores

3. Multi-Head Attention (MHA) 🎯

Instead of running attention once, we run it h times in parallel (multiple "heads"), each learning different types of relationships:

• Head 1: Might learn syntactic relationships (subject-verb agreement)
• Head 2: Might learn coreference (pronoun → noun it refers to)
• Head 3: Might learn semantic similarity (synonyms, antonyms)

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) × W_O
where head_i = Attention(Q × W_Q_i, K × W_K_i, V × W_V_i)

GPT-3 uses 96 attention heads with 128-dimensional each (96 × 128 = 12,288 total model dimension).

4. Architecture: Encoder-Decoder 🏗️

The original Transformer had two parts:

1. Encoder: Reads the input and builds rich contextual representations.
2. Decoder: Generates the output, attending to both its own previous outputs and the encoder's output.

• BERT (Encoder-only): Good for understanding (Classification, NER, Search). Can see all tokens bidirectionally.
• GPT (Decoder-only): Good for generation (Chatbots, Writing). Uses causal masking — each token can only attend to previous tokens.
• T5/BART (Encoder-Decoder): Good for transformation (Translation, Summarization).

5. Positional Encoding 📍

Since Transformers process all words at once, they don't inherently know the order of words. We must inject position information.

Absolute Positional Encodings (Original Transformer)

Add a sinusoidal vector based on position number:

PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Rotary Position Embeddings (RoPE) — Used by LLaMA, GPT-NeoX

RoPE encodes position by rotating the query and key vectors. This naturally captures relative positions rather than absolute ones.

• Advantage: Better generalization to sequence lengths not seen during training.
• Used by: LLaMA 2/3, Mistral, Falcon, GPT-NeoX, Phi.

📚 Official Resources

• RoPE Paper — Original Rotary Embeddings paper
• Jay Alammar: Illustrated Transformer — Best visual explanation

6. KV-Cache: The Secret to Fast Inference ⚡

During generation, the model processes the same past tokens repeatedly. KV-Cache stores the Key and Value matrices from previous steps so they don't need to be recomputed:

7. Flash Attention: Making It Practical at Scale ⚡

The bottleneck in standard attention is memory bandwidth — writing and reading the large N×N attention matrix from GPU HBM (high-bandwidth memory).

Flash Attention (Tri Dao et al., 2022) reorganizes the computation to keep intermediate results in the fast SRAM (on-chip cache):

• 2-4x faster training
• 10-20x less memory for the attention matrix
• Enables long contexts (32K, 128K, 1M tokens) that weren't previously possible

[!IMPORTANT] Flash Attention is now used by virtually all major LLM training runs. It's the reason models can have 128K+ token context windows.

📚 Official Resources

• Flash Attention Paper — Original Flash Attention
• Flash Attention 2 GitHub — Production implementation

Interactive Challenge: Attention Matrix

Imagine we have 3 words. The Attention Matrix shows how much each word focuses on others.

Quiz

Key Takeaways

✅ Self-Attention uses Q, K, V projections to let every token attend to every other token.
✅ Multi-Head Attention runs multiple attention operations in parallel, each learning different linguistic patterns.
✅ KV-Cache makes autoregressive generation efficient by storing past computations.
✅ Flash Attention tiles the attention computation to use fast on-chip memory, enabling 128K+ context windows.
✅ RoPE positional encodings (used in LLaMA, Mistral) generalize better than absolute sinusoidal encodings.

Official Resources

• 📖 "Attention Is All You Need" Paper
• 📖 Flash Attention Paper
• 📖 Flash Attention 2 GitHub
• 📖 Jay Alammar: Illustrated Transformer
• 📖 RoPE Paper

What's Next?

We have the building blocks. Now let's scale it up. Way up. Next Chapter: Large Language Models (LLMs) — Deep Dive.