The mathematics of attention

A complete derivation of scaled dot-product self-attention, built from the ground up. Defines queries, keys, and values as learned projections of the same input; shows how one attention score is computed and why it's scaled by the square root of the key dimension; turns scores into weights with a row-wise softmax and mixes values to produce the output; assembles the full equation and works a complete numeric example by hand; explains the causal mask and how position is injected through rotary embeddings; covers multi-head attention, self- versus cross-attention, why attention costs grow quadratically, what FlashAttention and sparse attention actually change, and what attention weights can and can't tell you about why a model produced its output.