Transformer Attention Shapes and Causal Masks Explained

An attention mask controls which token positions can contribute to an attention result. A causal mask hides future positions; a padding mask hides artificial padding. Both affect the scores used to combine value vectors.

Causal masks and padding masks solve different problems

A causal mask blocks positions to the right of the current token. A padding mask blocks placeholder tokens introduced to make a batch rectangular. A padded decoder batch can need both. Check the mask's broadcast shape against the attention scores, and confirm how the particular API interprets Boolean values.

In PyTorch scaled_dot_product_attention, a Boolean True indicates a position that participates in attention. Other interfaces can use True to mean masked out. An additive mask instead uses zero for permitted scores and negative infinity for blocked scores. Avoid leaving a query with every key masked, and test padded and unpadded examples separately.

From text to vectors

A tokenizer converts text into token IDs. An embedding table maps each ID to a vector. Tokens alone do not contain word order, so the model also needs positional information. The original transformer added sinusoidal position encodings; later models use learned positions or relative and rotary methods.

Tokenization affects cost and behavior. A "word" can become one token, several subword tokens, or bytes depending on the tokenizer. Context limits and API usage are measured in tokens, not characters.

Self-attention in one calculation

For every token vector, learned projections produce a query, key, and value. The query represents what the current position is looking for. Keys describe what each position can match; values hold the information that will be mixed.

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

Multiply queries by keys to produce a score between each pair of positions.
Divide by the square root of the key dimension to keep score magnitudes manageable.
Apply a mask when some positions must be hidden.
Use softmax to turn scores into weights.
Take a weighted sum of value vectors.

If the token "bank" appears near "river," attention can place weight on the surrounding words that support the geographic sense. Attention weights are intermediate computations, however, and should not automatically be treated as a complete explanation of the model's decision.

Why multiple heads help

Multi-head attention runs several smaller attention calculations with different learned projections, concatenates their outputs, and projects them back to the model dimension. Different heads can represent different useful relationships, but assigning a fixed human meaning to every head is usually too strong a claim.

The number of heads, head dimension, and model width are architectural choices. More heads do not guarantee better quality. They change capacity, memory access, and compatibility with optimized kernels.

The rest of a transformer block

Attention mixes information across token positions. A feed-forward network then transforms each position independently using the same learned layers. Residual connections provide short paths around both sublayers, and normalization stabilizes optimization. Dropout or related regularization may be used during training.

A simplified block is:

x = x + attention(normalize(x))
x = x + feed_forward(normalize(x))

This "pre-norm" form is common in later models; the original paper placed normalization differently. Repeating the block builds representations that integrate information over the sequence.

Encoder, decoder, and decoder-only models

Encoder: uses bidirectional self-attention, so each position can attend to the full input. It is well suited to representation and understanding tasks.
Decoder: uses causal masking in self-attention so a position cannot see future output tokens. In the original sequence-to-sequence transformer, it also attends to encoder output.
Decoder-only: stacks causal blocks and predicts the next token. Many general-purpose text generators use this form.
Encoder-decoder: remains useful when an input sequence is transformed into a distinct output sequence, such as translation or summarization.

During training, many next-token positions can be evaluated in parallel because the full target sequence is known and masked. During generation, each new token depends on earlier generated tokens, so decoding remains sequential at the outer loop.

Why long context is expensive

Standard self-attention creates a score for every pair of positions. Its attention matrix therefore grows quadratically with sequence length. A sequence twice as long has four times as many pair scores. Training also stores activations needed for backpropagation.

Generation uses a key-value cache so previous keys and values do not have to be recomputed at every step. The cache grows with context length, layer count, and key-value dimensions. Techniques such as grouped-query attention, sliding windows, sparse attention, and optimized attention kernels address different parts of this cost. They should be compared on target hardware and sequence lengths.

How a transformer learns

For causal language modeling, the model receives a token sequence and predicts the next token at each position. Cross-entropy loss penalizes probability assigned away from the observed next token. Gradient-based optimization updates embeddings, attention projections, feed-forward weights, and other parameters across many examples.

Instruction tuning and preference-based training can change how a pretrained model responds, but they do not turn generation into a database lookup. The model still produces probabilities over tokens. Retrieval, tools, validation, and explicit abstention are needed when an application requires current evidence or reliable actions.

A small shape example

Suppose a batch has 8 sequences, each 512 tokens, with model width 768 and 12 heads. The input representation has shape [8, 512, 768]. Each head works with a smaller dimension, commonly 64 in this example. Per head, the attention score tensor has shape [8, 512, 512]. Across 12 heads, those pairwise scores are a major memory cost even before gradients.

Shape reasoning is useful when reading model code. Many bugs come from mixing batch, sequence, head, and feature dimensions rather than from the attention formula itself.

What transformers are good at and where they fail

Transformers scale across text, vision, audio, protein sequences, and multimodal inputs because attention can connect elements in a flexible context. They also have limits: compute and memory grow with context, generation can be slow, outputs can be unsupported by evidence, and learned behavior reflects the data and objectives used in training.

For applications, architecture knowledge should lead to practical safeguards. Keep prompts within a tested context range, measure quality by task and subgroup, retrieve current sources when facts can change, validate structured outputs, and require approval before irreversible actions.

Frequently asked questions

Does attention understand meaning?

Attention computes learned weighted combinations of representations. The model can use those computations to capture useful linguistic and semantic patterns, but the matrix itself is not human understanding or a complete explanation.

Why are transformers trained in parallel but generated sequentially?

Training knows the target sequence and masks future positions, allowing many positions to be scored together. At generation time, the next token is unknown until the current step produces it.

Is a longer context window always better?

No. Longer context costs memory and compute, and models may not use every part equally well. Retrieval and concise context selection can outperform placing an entire document collection in one prompt.

Primary sources

A three-token causal mask you can check by hand

For three positions, the first may attend only to itself; the second may attend to positions one and two; the third may attend to all three. One additive mask is:

0    -inf -inf
0     0   -inf
0     0    0

Add it to the attention scores before softmax. Disallowed positions then receive zero weight. This is a mathematical illustration, not a promise that every attention API uses the same mask convention; check whether your API expects additive values or booleans.