LESSON 06 The repeatable unit
The small circuit a transformer runs again and again: mix context, preserve a route, stabilize the numbers, then enrich the signal.
↳ Short answer / layer by layer
Short answer: a transformer block is the repeated layer of a transformer. It takes a sequence of token vectors, lets each token gather context from the others with self-attention, adds the input back through a residual connection, normalizes the result, passes each position through a small feed-forward network, then adds and normalizes again. The output has the same shape as the input, so the next block can take it directly. Large language models are mostly a tall stack of transformer blocks.
This is the original layout from Vaswani et al. (2017), “Attention Is All You Need”, where normalization comes after each residual add (post-LN). Read it top to bottom. The same steps run in Experiment 01 below.
d_model per position. In the first block these are embeddings plus position information. In later blocks they are the previous block's output.d_model. See the attention lesson.x + Attention(x).d_model to d_ff, activate, project back. See the feed-forward lesson.d_model per position, ready for the next transformer block or, after the last one, the output head.In short: attention → add & norm → feed-forward → add & norm. Attention moves information between positions. The FFN transforms each position on its own. Residuals and normalization keep the signal stable across many stacked transformer blocks.
00 Overview / start here
A language model needs to keep a useful thread of meaning while it adds context and new features. The transformer block is the repeatable unit that does this work.
In “The robot carried the battery because it was heavy,” the token “it” starts with a rough representation. It needs the surrounding words, but the next calculation should not erase what “it” already carried.
It lets tokens exchange information with self-attention, adds the previous signal back through a residual path, stabilizes the scale with normalization, and then applies a small feed-forward network.
For this lesson, “it” is a four-number vector. The vector is a calculated state; the embedding table that produced it is learned.
Attention mixes information across positions. The feed-forward network works on each position separately to create richer features.
Residual additions keep a direct route from earlier values. The output can enter the next identical block, many times.
Attention pulls in a little “battery” and “heavy” information. The first residual adds the original “it” signal back. The FFN notices a feature combination, the second residual keeps both, and normalization keeps the values workable.
01 Trace the block
The names are less important than the sequence. Move stage by stage and watch the same four-number signal change. Use Previous, Next or the arrow keys when the stepper has focus.
This is a toy trace for the token “it.” Values are rounded to two decimals. The attention mix and FFN weights are illustrative learned parameters; residual addition and normalization are calculated from the current state.
A token begins as a row of learned numbers. Nothing has mixed yet.
A block does not replace the signal in one leap. It edits the signal in careful passes.
02 Run the skip-path experiment
A residual connection is an element-wise addition: line up two vectors and add their matching coordinates. Toggle it to see what the block loses when the direct route disappears.
The gauge below is deliberately simple: it shows how much of each original component is present in the first sum. It is not a probability and does not claim that a real model has a literal percentage of “old meaning.”
Each coordinate receives the old value and the new value. The block can add context without forcing the old route to vanish.
The shortcut is not laziness. It is a memory lane.
03 Repeat the unit
One block can make a small update. A stack of identical-shaped blocks can make a representation progressively richer. Change the depth and compare the toy states.
This control repeats the same illustrative block. It is not a trained language model: the weights stay fixed, the dimensions stay tiny, and the changing vectors only demonstrate repeated transformation.
| pass | vector state | what can change |
|---|
Depth is not a new kind of block. It is another chance to transform the same kind of state.
04 Do the small math
Now pin the moving picture to one explicit calculation. This makes the boundary clear: learned weights are parameters; the vectors that pass through the block are calculated states.
The left side is static so you can read it slowly. The right side mirrors the same numbers and labels the shapes, operation order and value origin.
x = [0.70, 0.20, 0.80, 0.10]0.15V₁ + 0.65V₂ + 0.20V₃
= [0.25, 0.47, 0.23, 0.54]x + attention
= [0.95, 0.67, 1.03, 0.64](z − mean(z)) / √(variance + ε)
≈ [0.73, −0.91, 1.23, −1.05]FFN(norm) ≈ [0.10, 0.15, −0.07, 0.16]
output ≈ [0.81, −0.92, 1.17, −1.06]04.1 Reference / residuals
A residual (skip) connection computes x + F(x), where F is the sub-layer (attention or FFN). The idea comes from residual networks (He et al., 2016). In a transformer block it does three jobs:
x and F(x) to be the same size, which is why every sub-layer outputs d_model features and why the whole block keeps its input shape.You can think of the residual stream as a running state that every block reads from and writes back to.
04.2 Reference / normalization
Where the layer norm sits relative to the residual add is one of the main ways transformer blocks differ.
| Post-LN (original) | Pre-LN | |
|---|---|---|
| Sub-layer step | x = LayerNorm(x + Sublayer(x)) |
x = x + Sublayer(LayerNorm(x)) |
| Where norm sits | After the residual add | Before each sub-layer, inside the branch |
| Residual path | Passes through a norm at every block | Stays an unnormalized straight path |
| Final layer norm | Not needed | Usually one extra norm after the last block |
| Training | Usually needs learning-rate warm-up | Xiong et al. (2020) showed it can train without warm-up |
| Used in | The original 2017 Transformer, BERT | GPT-2 and most modern decoder-only LLMs |
Xiong et al. (2020), “On Layer Normalization in the Transformer Architecture”, showed that with post-LN the gradients near the output layer are large at initialization, which is why a warm-up stage is needed. Moving the norm inside the branch (pre-LN) keeps them well behaved. GPT-2 (Radford et al., 2019) moved layer norm to the input of each sub-block and added a final layer norm after the last block. Many newer models keep the pre-norm layout but swap LayerNorm for RMSNorm, a simpler variant that skips mean-centering.
This lesson’s toy trace uses the post-LN order (normalize after each add) because it is easier to follow step by step.
04.3 Reference / shapes
Shapes are the fastest way to check that you understand a block. Use these names:
B = batch size, T = sequence length (tokens), d_model = vector width, h = number of heads, d_k = d_model / h = per-head width, d_ff = FFN hidden width.
The original base Transformer used d_model = 512, h = 8, d_k = 64 and d_ff = 2048, so the FFN is four times wider than the model.
(B, T, d_model)(d_model, d_model) → three tensors of (B, T, d_model)(B, h, T, d_k)Q · Kᵀ / √d_k → (B, h, T, T), one score for every pair of positions in every head. A causal mask goes here in decoder blocks.(B, h, T, T); each row sums to 1(B, h, T, d_k)(B, T, d_model)W_O of (d_model, d_model) → (B, T, d_model)(B, T, d_model)W₁ of (d_model, d_ff) + activation → (B, T, d_ff)W₂ of (d_ff, d_model) → (B, T, d_model)(B, T, d_model), the same as the inputTwo things to notice. First, the (T, T) score matrix is why attention cost grows with the square of the sequence length. Second, the FFN never mixes positions: W₁ and W₂ are the same for every token, so all cross-token communication happens in attention. The toy lesson above uses B = 1, T = 1 (the token “it” is traced) and d_model = 4.
04.4 Reference / block types
The original Transformer had two kinds of blocks. Modern models usually use one kind.
| Encoder block | Decoder block (original encoder–decoder) | Decoder-only block (GPT-style) | |
|---|---|---|---|
| Self-attention | Bidirectional: every token sees every token | Masked (causal): a token sees only earlier positions | Masked (causal) |
| Cross-attention | None | Yes: queries from the decoder, keys/values from the encoder output | None |
| Sub-layers | 2 (attention, FFN) | 3 (masked self-attention, cross-attention, FFN) | 2 (masked self-attention, FFN) |
| Typical use | Understanding text (BERT-style models) | Translation and other sequence-to-sequence tasks | Next-token generation (LLMs) |
The causal mask is what lets a decoder train on whole sequences at once while still predicting each token only from the tokens before it. For the different attention variants, see types of attention mechanism in transformers.
04.5 Reference / depth
Transformer blocks are stacked so each one refines the output of the one before (Experiment 03 lets you try this). Some published examples:
All the blocks in a stack have the same structure, but each has its own learned weights. For how the stack fits between tokenization and next-token probabilities, see how a transformer works.
05 Run the tiny version
A minimal JavaScript example performs one toy block in the same order as the diagrams. It runs in this page with no build step or library.
The code keeps the learned toy weights visible. Everything printed after `embedding` is calculated from them and from the current vector.
const embedding = [0.70, 0.20, 0.80, 0.10];
const attentionWeights = [0.15, 0.65, 0.20]; // toy learned scores
const values = [[0.20, 0.10, 0.40, 0.30], [0.30, 0.60, 0.20, 0.70], [0.10, 0.30, 0.20, 0.20]];
const W1 = [[0.60, 0.10, -0.20, 0.00], [-0.10, 0.40, 0.20, 0.30], [0.20, -0.30, 0.40, 0.10]];
const W2 = [[0.20, 0.00, 0.10], [-0.10, 0.10, 0.20], [0.10, 0.20, -0.10], [0.00, 0.30, 0.20]];
const add = (a, b) => a.map((value, i) => value + b[i]);
const mix = values[0].map((_, i) => values.reduce((sum, value, j) => sum + attentionWeights[j] * value[i], 0));
const normalize = (vector) => { const mean = vector.reduce((s, v) => s + v, 0) / vector.length; const variance = vector.reduce((s, v) => s + (v - mean) ** 2, 0) / vector.length; return vector.map(v => (v - mean) / Math.sqrt(variance + 0.0001)); };
const relu = value => Math.max(0, value);
const firstResidual = add(embedding, mix);
const stable = normalize(firstResidual);
const hidden = W1.map(row => relu(row.reduce((sum, weight, i) => sum + weight * stable[i], 0)));
const ffn = W2.map(row => row.reduce((sum, weight, i) => sum + weight * hidden[i], 0));
const output = normalize(add(stable, ffn));
console.log({ embedding, mix, firstResidual, stable, hidden, ffn, output });No run yet. The output will match the calculation above.
What this omits: batching, positional details, multiple attention heads, dropout, masking and training. Those are important production details, but hiding them here keeps the block’s causal order inspectable.
06 Consolidate / try this
Close the loop without memorizing the labels. Use the experiments above as evidence, then make one prediction.
Turn the residual connection off. Before you look at the output, predict what happens to the original vector’s component share and why the later normalization still runs.
Replace the attention weight for “robot” with 0.80 and lower the “The” weight. Which part of the attention mix gets louder? Which residual coordinates change?
Embeddings start the state. Attention mixes positions. Residuals preserve a direct route. Normalization keeps scale manageable. FFN enriches each position. The same shape can enter another block.
For the decoding control behind next-token variation, read what temperature means in an LLM. For the wider category around language models, see LLM vs. generative AI.
This field guide uses one token position, four features, fixed toy parameters and a simplified post-attention order. Real LLMs add positional information, multiple heads, causal masks, layer-norm variants, efficient kernels, batching, training updates and many more blocks. Those details refine the implementation; they do not change the core story traced here.
source note The original Attention Is All You Need paper introduced the Transformer architecture. This page’s vectors, weights and outputs are deliberately tiny teaching values. For normalization background, see the original Layer Normalization paper.
FAQ Transformer blocks / quick answers
Keep the reusable unit clear before the full transformer story gets larger.
A transformer block is a repeatable unit that lets positions exchange context through attention, updates features with a feed-forward network, and uses residual paths and normalization to keep the signal workable.
A residual connection adds an earlier vector back to a transformed vector element by element. It gives the block a direct route for preserving useful information while adding a new update.
Each block can refine the representation it receives. Repeating the same overall shape lets later blocks work with a state that already contains more context and feature transformations.
It varies by model. The original Transformer used 6 in the encoder and 6 in the decoder, BERT-base uses 12, the largest GPT-2 uses 48, and GPT-3 175B uses 96. More blocks add capacity and compute cost.
Usually, yes. Papers and code often say “layer” for one whole block (attention + FFN with their residuals and norms). Be aware that “layer” can also mean a single sub-layer or a single linear layer, so check the context.
Post-LN normalizes after each residual add, as in the original 2017 Transformer. Pre-LN normalizes the input to each sub-layer and leaves the residual path untouched. Pre-LN is common in modern LLMs because it trains more stably and, as Xiong et al. (2020) showed, can work without learning-rate warm-up.
The FFN expands each vector to d_ff (four times d_model in the original Transformer), applies a non-linearity, then projects back. The wider hidden layer gives the block room to compute new feature combinations for each position before writing a d_model-sized update back into the residual stream.