A summary still needs processing.
Imagine attention has already gathered clues about the word “bank.” The model now needs to turn that bundle into a sharper, more useful representation for the next layer.
Back to the lesson shelf
Feed-forward
LESSON 05 The private workshop
Feed-forward
network.
A small two-stage workshop that reshapes one token representation without looking at its neighbors.
In one lineExpand → activate → compress. Same rule, one token at a time.
POSITION_WISE_FFN / 05 reshaping
one token innew signal out
input → hidden features → outputsame weights / local work
Short version: attention gathers; the feed-forward network reshapes.Step, change, compare.
00 Overview / start here
After context comes transformation.
Attention can bring useful information into a token. A feed-forward network then works on that token’s representation alone, turning combinations of features into a new signal.
A private workshop for each token.
A feed-forward network applies the same two learned linear transformations and a nonlinear activation to each token separately. No token can hand information to its neighbor inside this operation.
the basic storyinput → operation → output
- 01 / enterOne vector arrives.
A token representation, such as
[0.80, 0.20, 0.60], enters. - 02 / expandMore feature slots.
A linear layer makes a wider hidden vector so several combinations can be inspected.
- 03 / activateKeep useful signals.
ReLU can set negative signals to zero; GELU bends them smoothly instead.
- 04 / compressReturn to model width.
A second linear layer mixes hidden features back into the model-sized vector.
- 05 / leaveA changed representation.
The output continues through the transformer block, ready for the next operation.
Workshop, not a lookup table.
The input is a bundle of ingredients, hidden neurons are temporary workbenches, and the output is a repacked bundle. The limit: neurons do not have human names like “meaning” or “grammar”; those labels are teaching shorthand, not a readable dictionary.
mental checklist
Ask four questions.
- What vector enters?
- How wide is the hidden workspace?
- Which activation changes the signal?
- What vector leaves?
01 One token, five states
Make the chain visible.
Choose a signal, change the hidden capacity or activation, and step through the same token from input to output. The weights stay fixed so you can see which control caused each change.
Run the private workshop.
The default vector is [0.80, 0.20, 0.60]. Its shape is [3]: three features for one token. The hidden workspace has shape [4] by default, and the output returns to shape [3].
input signal / feature balance
The three numbers are feature strengths, not words. Switch one balance to see different neurons respond.
activation / what happens to a signal?
ReLU keeps positive values and turns negative values into exactly 0. That clear zeroing makes the boundary easy to inspect.
01 / inputx
Token representation
Three feature values arrive from the earlier transformer work.
shape [3] / calculated upstream02 / expandz = W₁x + b₁
Pre-activation
Each hidden neuron forms a weighted sum before the gate.
shape [4] / learned W₁, calculated z03 / activateh = φ(z)
Gated features
Activation changes the pre-activation values one by one.
shape [4] / calculated h04 / compressy = W₂h + b₂
Output projection
The wide hidden signal is remixed into model width.
shape [3] / learned W₂, calculated y05 / outputnext layer
A new token signal
The vector is ready to meet the residual path and continue.
same values / new role in the blockhidden neurons / feature barswidth 4 / activation: ReLU
Neuron 3 is silent because its pre-activation is negative. ReLU turns that value into 0.
The hidden width is a workspace, not a second sentence. One token enters; one token leaves.
02 Same rule, different token
One workshop. Many tokens.
Transformers reuse the same feed-forward weights at every position. Click a token below: only the input vector changes. The rule, hidden width, and activation stay the same.
Watch the private rule repeat.
This is the contrast with attention: attention mixes information across token positions; the feed-forward step applies the same local transformation independently to each position.
input vector
same rule ↓
hidden activation
same rule ↓
output vector
Attention asks, “which positions should contribute?” The feed-forward network asks, “how should this one representation change?”
03 Do the small math
Every bar has a receipt.
The visualization is not guessing. Each hidden value is a sum of three products, and each output value is a sum of hidden features multiplied by a second row of weights.
Read one neuron from left to right.
The table starts from the default bank-like vector, width 4, and ReLU. Learned weights are shown in blue-gray; values calculated from them are live.
input x[0.80, 0.20, 0.60]
W₁ row / neuron 1[1.00, −0.50, 0.25]
bias b₁0.00
| hidden neuron | weighted sum before activation | activation |
|---|
second projection / W₂h[0.95, −0.06, 0.47]
y₁ = 0.70×0.85 + 0.20×0.26 + −0.10×0.00 + 0.40×0.76 = 0.95
shapes in this toy
x has shape [3]; W₁ has shape [4 × 3]; h has shape [4]; W₂ has shape [3 × 4]; y has shape [3]. Matrix dimensions tell us which multiplication is legal and what width comes out.
A neuron is not a tiny person with a label. It is a weighted sum followed by a shared rule.
04 Make it runnable
The whole mechanism in plain JS.
Once the states are clear, the code is short. There is no framework here: arrays hold vectors, loops compute dot products, and the activation is one ordinary function.
Input → expand → activate → compress → output.
The snippet uses the same default numbers as the calculation above. Run it in the page and compare the observed output with the receipt.
feed-forward.js / smallest runnable core
const input = [0.8, 0.2, 0.6];
const expand = [
[1.0, -0.5, 0.25],
[-0.3, 1.0, 0.5],
[0.6, 0.2, -1.0],
[0.1, 0.7, 0.9],
];
const preActivation = expand.map((row) =>
row.reduce((sum, weight, index) => sum + weight * input[index], 0),
);
const hidden = preActivation.map((value) => Math.max(0, value));
const compress = [
[0.7, 0.2, -0.1, 0.4],
[-0.4, 0.8, 0.6, 0.1],
[0.2, -0.3, 0.9, 0.5],
];
const output = compress.map((row) =>
row.reduce((sum, weight, index) => sum + weight * hidden[index], 0),
);
console.log(output.map((value) => Number(value.toFixed(2))));observed outputThis matches the hand calculation: the code is doing the same two matrix-shaped passes.
what enters
input is one token vector with three values.
what acts
expand computes four dot products, ReLU changes them, then compress computes three more dot products.
what leaves
output is a new three-value vector. The example omits batching, residual connections, normalization, and training.
WHAT IS DEFERRED / KEEP THE CORE CLEAR
Useful details, later.
Real transformer feed-forward blocks add engineering and training context that this field guide intentionally leaves out. The core causal chain stays the same.
The block usually adds the old vector back.
Production transformer layers commonly wrap the sub-layer in residual connections and layer normalization.
Many tokens can be computed in parallel.
“One token at a time” describes independence, not a slow loop. Hardware can process the whole sequence batch together.
The weights are learned from error.
Here the numbers are fixed for inspection. Training changes them so outputs become useful for the task.
TRY IT YOURSELF / CONSOLIDATE
Predict before you click.
Start with the default token. First predict what happens to neuron 3 when you switch from ReLU to GELU. Then choose “edge-heavy,” set width to 6, and explain which stage changed the vector’s shape and which stage changed its values.
With ReLU, neuron 3 is exactly 0 because its pre-activation is −0.08. GELU keeps a small negative value instead of hard-zeroing it, so the hidden vector changes smoothly. Width changes how many hidden slots exist; activation changes the values inside those slots.
PRIMARY SOURCE The original Transformer paper places a position-wise fully connected feed-forward network beside multi-head attention in each layer. Its position-wise design is the basis for this lesson’s contrast: attention relates positions, while this sub-layer transforms each position independently. Read section 3.1 of Attention Is All You Need.
FAQ Feed-forward networks / quick answers
A private
workshop.
Three quick answers about the part of a transformer that reshapes features locally.
01 / definitionWhat does a feed-forward network do in a transformer?
It transforms each token’s current vector independently: a first matrix expands the features, an activation changes the response, and a second matrix compresses them back to model width.
02 / shapeWhy expand, activate and compress?
The wider hidden space gives the network more room to form feature combinations. The nonlinear activation keeps the two linear steps from collapsing into one simple linear transformation.
03 / positionsDoes an FFN mix information between tokens?
No. The same weights are reused at every position, but the operation works on one position’s vector at a time. Attention is the part that mixes information across positions.