The Evolution of Attention

Used in: Original Transformer, early GPTs.

The Math Change: Instead of doing the core formula once, we split Q, K, and V into multiple smaller chunks (heads). We run the exact same formula independently on each chunk, then glue (concatenate) the results together.

Simple Explanation: One head might focus on grammar (verbs/objects), while another focuses on pronouns, and another focuses on context. Running the math in parallel lets the model capture different "relationships" at the same time.

Used in: Llama 2/3, PaLM.

The Math Change: Instead of having unique K and V for every attention head, multiple Q heads are forced to share the same K and V.

Simple Explanation: The math stays identical, but we dramatically shrink the size of the K and V matrices. This means the model uses far less memory when generating text, allowing it to run much faster, at a tiny cost to its "focus" diversity.

Used in: Image-to-text models, translation (old Transformers), stable diffusion.

The Math Change: The Q comes from one source (e.g., the text you are generating), but the K and V come from a completely different source (e.g., the image pixels or the foreign-language sentence).

Simple Explanation: You are asking a question (Q) about a specific database of facts (K, V). The math doesn't change, but where the numbers come from changes entirely. It lets a model read a document and answer questions about it.

Used in: Longformer, Mistral 7B, BigBird.

The Math Change: Instead of calculating $QK^T$ for every single token in a 10,000-word document (which is mathematically $O(n^2)$ and impossibly slow), we force the scores for far-away tokens to be zero. We only calculate the dot product for local neighbors (sliding window) and a few special "global" tokens.

Simple Explanation: Instead of comparing every word to every other word (which is math-heavy), we tell the model: "Only look at the words right next to you, plus a few main topic words." The math is the same, but we skip 95% of the calculations to handle huge documents.

Used in: Performer, RWKV (partially), some efficient models.

The Math Change: This is a big math trick. Normally, you do $(Q \times K^T) \times V$. Mathematically, this forces you to build a massive matrix. Linear attention uses a math property called the kernel trick. It replaces the dot-product with a special function (like $\text{elu}(x)+1$) so that you can rewrite the math as $Q \times (K^T \times V)$.

Simple Explanation: By changing the order you multiply the numbers, you never have to build the giant "similarity matrix." This drops the math workload from $O(n^2)$ down to $O(n)$ (linear). The trade-off is that it loses some precision on very long sequences, but it is lightning-fast.

Used in: Almost every modern model (GPT-4, Llama 3, Mistral).

The Math Change: Here is the secret: Flash Attention does NOT change the math formula at all. It calculates $\text{Softmax}(QK^T)V$ to the exact decimal as the original formula.

Simple Explanation: Instead of calculating all the scores, writing them to temporary memory, reading them back, and then doing Softmax, Flash Attention does the math in tiny overlapping chunks (tiling) directly inside the GPU's super-fast cache memory (SRAM). It uses smart math (online softmax) to stitch these chunks together. The result is the exact same answer, but it runs 2x–10x faster and uses far less memory.