FIELD GUIDE
Types of Attention Mechanism in Transformers: Self, Cross and Causal Attention
Compare self-attention, cross-attention and causal attention in transformers by their Q, K and V sources and which tokens they can see.
The main types of attention mechanism in transformers are easiest to distinguish by asking two questions: where do the query, key and value vectors come from, and which positions can each query see? Self-attention uses one sequence as its source. Cross-attention uses a query from one stream and keys and values from another. Causal attention limits visibility, usually hiding future positions in a decoder. These are related design choices, not three mutually exclusive boxes.
| Attention view | Query (Q) source | Key (K) and value (V) source | What it controls |
|---|---|---|---|
| Self-attention | A sequence representation | The same sequence representation | How positions in one stream use context from other allowed positions |
| Cross-attention | A query sequence | A separate context or memory sequence | How one stream retrieves information from another |
| Causal or masked attention | Usually the current sequence | Usually that same sequence | Which positions are visible; a causal mask blocks future positions |
| Multi-head attention | Learned projections of Q | Learned projections of K and V | Runs several attention calculations in parallel; can be used with self- or cross-attention |
The original Transformer paper describes self-attention in the encoder, masked self-attention in the decoder, and encoder-decoder attention using decoder queries with encoder keys and values. Vaswani et al., “Attention Is All You Need”
Self-attention: one stream supplies Q, K and V
In self-attention, the query, key and value vectors are computed from the same stream of representations. A word can use information from other words in that sequence to build its updated representation. In an encoder layer, a token may attend to other positions across the input, subject to any mask or implementation limit.
For example, in “The dog dropped the ball because it was tired,” the representation for “it” can use surrounding words as context. Self-attention describes the source relationship between Q, K and V; it does not guarantee that a particular head will resolve a pronoun correctly.
Cross-attention: query one stream with another
Cross-attention connects two streams. Queries come from the stream being updated, while keys and values come from a separate context. In the original encoder-decoder Transformer, decoder queries look across the encoder output. During translation, the decoder can use the source-language representation while forming the next target-language representation.
The practical clue is a difference in source: Q asks what this position needs; K and V provide the context it can retrieve. This is why cross-attention often appears in encoder-decoder models and multimodal systems that connect text with image or audio features.
Causal attention changes visibility
A causal mask blocks a position from using future positions. In an autoregressive text decoder, that keeps the model from seeing the next token while predicting it. The query, key and value vectors can still come from the same sequence, so causal attention is commonly causal self-attention rather than a separate source arrangement.
Masks can also represent other restrictions, such as padding or a local window. The PyTorch scaled-dot-product attention documentation exposes a causal option, and its MultiheadAttention documentation describes applying a causal mask. The exact mask and tensor layout depend on the implementation.
Multi-head is another architectural dimension
Multi-head attention projects Q, K and V into several learned subspaces, applies attention in parallel and combines the results. It is not a category that excludes self-attention or causal attention: a model can use multi-head causal self-attention, or multi-head cross-attention. In the original paper, the Transformer uses multi-head attention for encoder self-attention, decoder masked self-attention and encoder-decoder attention.
The common mistake is to compare “multi-head” with “self” as if they described the same property. Self versus cross tells you where the input streams come from. Causal versus unmasked tells you which positions are visible. Multi-head tells you how many projected attention computations run in parallel.
You may also see the unhyphenated phrases self attention, cross attention, causal attention and multi head attention in search queries or notes; they refer to the same distinctions described above.
Quick way to identify the mechanism
When reading a model diagram or code, trace the tensors rather than relying on a label. If Q, K and V all originate in one stream, it is self-attention. If Q comes from one stream and K/V from a separate memory, it is cross-attention. Then inspect the mask to see whether future or otherwise restricted positions are hidden. Finally, check whether the operation uses one head or several.
Continue with the site's interactive lessons on attention, the transformer block and how transformers work.
FAQ
Is causal attention the same as self-attention?
No. Causal describes a visibility rule; self describes where Q, K and V come from. Causal masking is commonly applied to self-attention in a decoder.
Is multi-head attention a separate type from cross-attention?
No. Multi-head describes parallel learned projections. Those heads can be used for self-attention or cross-attention.
Does every transformer use cross-attention?
No. Encoder-decoder Transformers use cross-attention between decoder and encoder streams. Decoder-only architectures can operate without an encoder context stream.