1Neural Networks 2The Perceptron 3Forward Propagation 4Activation Functions 5Bias & Decision Boundaries 6Softmax, Logits & Temperature 7Loss Functions 8Backpropagation

Lensa ML

How Forward Propagation Works

Step 1 of 6: Raw Features Go In

The network sees numbers, not pictures — each feature is a measurement

A cat image enters the network

Pointy ears

0.85

normalized 0–1

Whiskers

0.72

normalized 0–1

Fur texture

0.91

normalized 0–1

The network doesn't see a photo. It sees numbers: each feature is a measurement scaled to 0–1, and a high value means "strongly present." These three numbers are all the network has to work with.

① Measure features

② Normalize to 0–1

③ Feed into layer 1

Watch the three feature cards fill in: Pointy ears 0.85, Whiskers 0.72, Fur texture 0.91. Together they form the input vector x = [0.85, 0.72, 0.91]. A real image model uses far more inputs; a 28×28 grayscale digit already gives 784 pixel values. Scaling inputs to a similar range matters because a feature measured in thousands would dominate every weighted sum and push sigmoid neurons into their flat regions. In practice, frameworks process many examples at once by stacking their input vectors into a matrix, one row per example.

Step 2 of 6: Weights Shape the Signal

Every connection multiplies the input by a learned weight

Tap a neuron to see its weights

Every connection has a weight, a number that amplifies, dampens, or flips the signal. Tap any neuron to see the weights feeding into it.

Tap a neuron in a hidden or output layer to list the weights feeding into it. The weights between two layers form a matrix. Here the input-to-Hidden 1 matrix is 3×4, with one column per Hidden 1 neuron, so the whole layer is computed as one matrix product x·W. That is why GPUs, which are built for matrix multiplication, speed up neural networks so much. These weights start between −0.8 and 0.8. Real networks scale their starting weights by layer size, as in Xavier or He initialization.

Step 3 of 6: Neurons Sum It Up

Each neuron collects weighted signals and adds them together

InputValue×Weight=Product

Pointy ears...×-0.57=...

Whiskers...×+0.48=...

Fur texture...×+0.68=...

Σ sum =...

z = ?×(?) + ?×(?) + ?×(?) = ...

Each neuron is a collector: it adds every weighted input into one number, the weighted sum (z). The raw sum can be any real number, positive or negative. The activation function comes next.

Click H1#1 through H1#4 to compute each neuron's weighted sum. For H1#1: z = 0.85×(−0.57) + 0.72×0.48 + 0.91×0.68 = −0.485 + 0.346 + 0.619 = 0.480. This is a dot product, so z is large when the input vector points in the same direction as the neuron's weight vector. Most networks also add a bias, z = w·x + b. This network leaves the bias out so every sum can be checked by hand.

Step 4 of 6: The Activation Function

Sigmoid squashes any number into a smooth 0–1 range

z value

0.48→0.6177

z ≪ 0

Output → 0

z ≈ 0

Output ≈ 0.5

z ≫ 0

Output → 1

The sigmoid function squashes any number into the 0–1 range. That gives the neuron a smooth on/off switch: instead of a hard 0 or 1, it outputs a graded value.

Drag the z value slider from −6 to 6 and read the output. The sigmoid is σ(z) = 1 / (1 + e^(−z)). At z = 0.48, the sum from H1#1, σ gives 0.618, which is that neuron's output in the next step. At the slider's ends the output is 0.0025 and 0.9975. Sigmoid is a natural fit for outputs that read as probabilities. Hidden layers in modern networks usually use ReLU, max(0, z), which is cheaper to compute and does not flatten out for large positive z.

Step 5 of 6: Layer by Layer

Watch data ripple forward through the entire network

Data flows forward through the network

Input

avg: 0.827

Hidden 1

—

Hidden 2

—

Output

—

Each layer applies weights → sum → sigmoid and passes the result forward. In a trained network, features grow more abstract with each layer, from "pointy ears" toward "cat-like." Values only move forward; no layer looks back.

Click a layer label to stop the animation and inspect it, or let it cycle. Written out, the whole pass is h₁ = σ(xW₁), h₂ = σ(h₁W₂), and y = σ(h₂W₃): a composition of three functions. Each layer needs the previous layer's full output, so layers run in order. Neurons within one layer do not depend on each other and can run in parallel. The average activation rises from about 0.52 in Hidden 1 to about 0.58 in Hidden 2. The pass costs 36 multiply-adds, one per weight.

Step 6 of 6: The Prediction

The output layer speaks — the network has made its guess

① Inputs enter

② Weights shape signals

③ Layers transform

④ Output predicts

Look at the two output cards: Cat 0.57 and Dog 0.67. The untrained network calls this cat a dog. The cards call them sigmoid scores, not confidences: each output is a separate sigmoid, and nothing has calibrated those scores to match real frequencies. Training starts by comparing the outputs with the targets Cat = 1 and Dog = 0. The sum of squared errors here is (1 − 0.571)² + (0 − 0.671)² ≈ 0.63. Backpropagation then traces that loss back to the weights.

What is forward propagation in a neural network?

Forward propagation is the computation a neural network runs to turn an input into a prediction. Each layer takes the previous layer's outputs, multiplies them by its weights, adds them up, and applies an activation function. The result flows to the next layer until the output layer produces the answer.

Every use of a trained model is a forward pass: classifying an image, scoring a loan application, or generating the next word in a chatbot reply. Training runs the forward pass too, because the loss and the gradients are both computed from its outputs. The example here follows one cat photo through a network with 3 inputs, two hidden layers of 4 neurons, and 2 outputs.

Key ideas

A forward pass turns an input vector into a prediction, one layer at a time.
Each neuron computes a weighted sum of its inputs and applies an activation function.
A whole layer is one matrix multiplication followed by an element-wise activation.
Layers must run in order, but neurons within a layer can run in parallel.
With random weights, the forward pass still produces an answer, just a poor one.
Training uses the forward pass's outputs to compute the loss and the gradients.

Common questions

What is the difference between forward propagation and backpropagation?

Forward propagation computes the prediction from the input. Backpropagation runs after it, starting from the loss and moving backward to compute each weight's gradient. Inference needs only the forward pass; training needs both.

Is forward propagation the same as inference?

Inference is running a trained model on new data, and it consists of forward passes. During training, a forward pass is also run on every batch of training examples, so forward propagation is used in both phases.

Why do activations need to be stored during the forward pass?

Backpropagation needs each layer's inputs and outputs to compute the gradients. During training, the framework keeps them in memory until the backward pass is done. At inference time nothing is stored, which is why running a model takes much less memory than training it.

How is forward propagation done for a batch of inputs?

The input vectors are stacked into a matrix X with one row per example. Each layer then computes σ(XW + b) in one matrix multiplication, producing outputs for every example at once. Batching makes far better use of GPU hardware than processing examples one by one.

Related explainers

Neural Networks — Neurons, layers, and how networks learn
Activation Functions — Why networks need non-linearity to learn
Backpropagation — How networks learn from their mistakes
Weight Initialization — Xavier, He, and why random weights matter
Tensors, Broadcasting & Reshaping — The data structures of deep learning
Loss Functions — How networks measure their mistakes

References

Last updated October 25, 2026