Build a neural
network from scratch.

What this tutorial builds

We will implement a small feed-forward neural network in Python with NumPy. It learns the XOR truth table, a compact example a single perceptron cannot solve with one straight decision boundary.

You need basic Python and the idea of a function. Install NumPy with python -m pip install numpy. NumPy’s quickstart explains the arrays and matrix multiplication used below.

Why a single perceptron misses XOR

For XOR, the output is 1 when exactly one input is 1: (0, 1) and (1, 0). It is 0 for (0, 0) and (1, 1). No single straight line separates those two positive examples from the two negative ones. A perceptron can learn linearly separable patterns such as AND or OR, but it needs hidden units to express XOR.

The network below has two input values, one hidden layer and one output. It is still small enough to follow by hand.

One compact NumPy implementation

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

def sigmoid(z):
    z = np.clip(z, -500, 500)
    return 1.0 / (1.0 + np.exp(-z))

rng = np.random.default_rng(7)
W1 = rng.normal(0.0, 0.5, size=(2, 4))
b1 = np.zeros((1, 4))
W2 = rng.normal(0.0, 0.5, size=(4, 1))
b2 = np.zeros((1, 1))
learning_rate = 0.5
samples = len(X)

for epoch in range(30000):
    # Forward pass
    hidden = sigmoid(X @ W1 + b1)
    probabilities = sigmoid(hidden @ W2 + b2)

    # Backward pass: sigmoid output + binary cross-entropy
    output_error = (probabilities - y) / samples
    grad_W2 = hidden.T @ output_error
    grad_b2 = output_error.sum(axis=0, keepdims=True)

    hidden_error = (output_error @ W2.T) * hidden * (1.0 - hidden)
    grad_W1 = X.T @ hidden_error
    grad_b1 = hidden_error.sum(axis=0, keepdims=True)

    # Gradient-descent update
    W2 -= learning_rate * grad_W2
    b2 -= learning_rate * grad_b2
    W1 -= learning_rate * grad_W1
    b1 -= learning_rate * grad_b1

predictions = sigmoid(sigmoid(X @ W1 + b1) @ W2 + b2)
print((predictions >= 0.5).astype(int).ravel())

For the four teaching rows, the final line should classify the inputs as [0 1 1 0]. The exact probabilities depend on the learned weights; the threshold turns them into labels for this demonstration.

X @ W1 is matrix multiplication: NumPy’s @ operator combines each row of inputs with the first layer’s weights. Biases are then added, and sigmoid maps each value to the 0–1 range. See the official NumPy quickstart for array shapes and matrix products.

1. 01 / forwardCalculate activations.

   The hidden values are sigmoid(X @ W1 + b1); the output is another weighted sum followed by sigmoid.

2. 02 / compareFind output error.

   With sigmoid plus binary cross-entropy, the derivative at the output simplifies to (prediction - label) / samples.

3. 03 / chainCarry the error back.

   The hidden-layer derivative multiplies by the output weights and by the sigmoid derivative, a × (1 - a).

4. 04 / updateMove parameters against the gradient.

   Subtract learning rate times each gradient. Repeating the loop should reduce the loss for this small, well-defined example.

W1 has shape 2 × 4, b1 has shape 1 × 4, W2 has shape 4 × 1, and b2 has shape 1 × 1. A shape mismatch usually means an input, layer width or transpose does not match the equation.

Common beginner mistakes

• Confusing element-wise multiplication * with matrix multiplication @.
• Updating weights before computing all gradients for the current step.
• Assuming a low training loss proves performance on new examples.
• Changing the threshold without saying what decision it represents.
• Applying a tutorial network to people or consequential decisions without suitable data and review.

For a next step, compare this hand-coded model with the neural-network field lesson, then explore gradient descent or choose a carefully scoped idea from beginner machine learning projects. The beginner course explains the difference between training, validation and test data.