Study

Self-attention lets every token in a sequence compare itself with every other token. For each pair, the model computes a compatibility score, normalises the scores with a softmax into weights, and returns a weighted average of the value vectors. The result is a contextual representation: the same word gets a different vector depending on its neighbours.

Attention weights answer 'how much should this token listen to each other token'. They do not encode order by themselves. If you permute the input tokens and permute the outputs the same way, the computation is unchanged: self-attention without position information is permutation-equivariant.

Each token is projected into three vectors: a query (what this token is looking for), a key (what this token offers to others), and a value (the content carried forward). The attention score between token i and token j is the dot product of query i and key j, scaled by the square root of the key dimension.

Queries ask, keys advertise, values deliver. After softmax, the weights multiply the values: output_i = sum_j weight_ij · value_j. Two tokens can therefore have similar keys (both match a query) but carry different values (contribute different content).

A standard Transformer has no recurrence and no convolution, so by itself it cannot tell first from last: without positional information it does not know token order. Positional encodings (fixed sinusoidal patterns or learned vectors) are added to the token embeddings so that 'dog bites man' and 'man bites dog' produce different representations.

Because attention is permutation-equivariant, removing positional information makes the model treat a shuffled sentence nearly the same as the original: it can still compare tokens and find which ones are related, but it cannot distinguish their order. Exam questions on this point ask what property of the sequence is lost — the answer is order, not importance or relevance.

Multi-head attention runs the query-key-value computation h times in parallel, each in a smaller subspace. Different heads specialise: one may track subject-verb agreement, another coreference, another positional adjacency. Their outputs are concatenated and projected back to the model dimension.

A single attention head computes one weighted average, which bottlenecks what it can express. With eight heads, the model computes eight different averages and combines them, so each layer captures several relation types at once instead of one.

Backpropagation applies the chain rule to compute the gradient of the loss with respect to every weight. Gradient descent then steps each weight against its gradient, scaled by the learning rate. The learning rate is a step-size choice, not part of the gradient computation itself.

Backpropagation computes gradients; gradient descent uses them. Confusing the two is the most common error in this section: backprop answers 'which direction', the optimiser answers 'how far'.

Policy iteration alternates two steps: policy evaluation (compute values for the current policy) and policy improvement (act greedily with respect to those values). Value iteration instead applies the Bellman optimality update directly at every sweep, folding improvement into the update.

Policy evaluation estimates how good the current behaviour is; policy improvement changes the behaviour. Value iteration skips the full evaluation and always updates toward the optimal value function. Mixing up 'improvement' with 'value iteration' signals a policy/value confusion.