Lensa ML
Lensa ML

Bias & Decision Boundaries

Step 1 of 6: What Is Bias?

The shift in a neuron's output

0.5σ(0+b) = 0.500x
σ(x + b)σ(x), no bias
Biasb = 0.0
Bias
0.0
σ(0+b)
0.500
Shift
none

The bias shifts the sigmoid curve horizontally. At b = 0, the sigmoid crosses 0.5 exactly at x = 0. Move the slider to shift that crossing point.

Drag the bias slider between −5 and 5 and compare the shifted curve with the dashed one. A neuron σ(wx + b) crosses 0.5 where wx + b = 0, that is at x = −b/w. Here w = 1, so the crossing sits at x = −b. At b = 5 the neuron outputs 0.993 with zero input, so the bias also sets a neuron's default output when its inputs are all zero. Biases usually start at zero, and some ReLU networks start them slightly positive so every neuron is active at first.

Step 2 of 6: Without Bias

Boundaries must pass through the origin

-3-2-1123x₁x₂origin (0,0)
Rotate boundary45°
Accuracy
73%
Angle
45°
Constraint
Through (0,0)

Without bias, the boundary must pass through the origin. You can only rotate it, never shift it. The clusters sit off-center, so no angle gets every point right. At this angle 11 points sit on the wrong side, and the line can't slide over to fix them.

Drag Rotate boundary from 0° to 180°. The line's normal is (cos θ, sin θ), and it always passes through the origin. This dataset is two clusters centered at (−0.5, 0.5) and (2.5, 1.5), off to one side of the origin. The best angle, near 0°, gets 85%, and 45° gets 73%. A vertical line at x₁ = 1 would get every point right, but reaching it means shifting the line, which only a bias can do. Centering input data, by subtracting each feature's mean, is one reason a missing or poorly set bias hurts less in practice.

Step 3 of 6: With Bias

Shifting the boundary freely

-3-2-1123x₁x₂
w₁+1.00
w₂+0.50
bias0.00
With bias
80%
No bias
80%
Improvement
+0

With bias, the boundary can translate freely. The dashed line shows where the boundary would be without bias. Adjust the bias to shift the boundary and improve accuracy.

Set w₁, w₂, and bias, and compare the solid boundary with the dashed one through the origin. At the starting weights (1, 0.5), both lines score 80%; moving the bias to about −1.2 raises it to 39 of 40. During training the bias gets its own gradient, and because ∂z/∂b = 1, that gradient equals the neuron's error signal. Biases are a tiny share of a network's parameters, and weight decay is often switched off for them because shrinking a bias does not make the model simpler.

Step 4 of 6: Bias in 2D

Two inputs, one boundary

-3-2-1123x₁x₂1.2x₁ + 0.8x₂ − 1.5 = 0
w₁+1.2
w₂+0.8
bias-1.5
Accuracy
95%
Correct
38/40
Misclassified
2

In 2D, the boundary is the line w₁x₁ + w₂x₂ + b = 0. The weights control the line's angle, and the bias controls its position. The green and red regions show where the neuron outputs 1 vs 0.

Adjust w₁, w₂, and bias between −3 and 3 and watch the shaded regions follow the line. The starting values classify 38 of 40 points. For any point x, the value (w·x + b) / ‖w‖ is its signed distance from the boundary: positive on the green side, negative on the red. The neuron's raw sum is that distance times ‖w‖, so larger weights make the output change faster near the line. With three inputs the boundary becomes a plane, and with d inputs it is a (d − 1)-dimensional hyperplane.

Step 5 of 6: Multiple Neurons

Combining boundaries

-3-2-1123x₁x₂
Hidden neuron:123
Hidden neurons3
Neurons
3
Boundaries
3
Accuracy
98%

Green points sit inside a ring of red ones, so no single line can separate them. With 3 neurons, the network creates 3 decision boundaries, and 1 of 48 points still land on the wrong side. Add a neuron to tighten the fence. The shaded regions show the combined output: the output neuron merges the hidden neurons into one non-linear decision region.

Change Hidden neurons from 2 to 5 and toggle Show Combined Region. Here the green class sits inside a ring of red points, which no single line can separate. Each hidden neuron fires on the inner side of one line, and the output neuron fires only when all of them do, so the lines fence in a polygon. Accuracy goes 92%, 98%, then 100% at four and five neurons. The lines face evenly spaced directions, and only their shared distance from the center is fitted. Training would place every line freely. In a ReLU network, the hidden boundaries cut the input space into many convex regions, and the network computes a different linear function inside each one.

Step 6 of 6: The XOR Solution

Two boundaries, one answer

-3-2-1123x₁x₂One boundary cannot solve XOR
Layers
1
Accuracy
55%
Solvable?
No

XOR is not linearly separable: no single straight line separates the green from the red points, so a one-layer perceptron fails. Yellow-ringed points are misclassified.

Switch between 1 Layer (Perceptron) and 2 Layers (MLP). The single line x₁ + x₂ = 0 gets 55% of the 40 points. The network uses hand-set weights with a gain of 10: h₁ = σ(10x₁) and h₂ = σ(10x₂), then AND = σ(10(h₁ + h₂ − 1.5)) and OR = σ(10(h₁ + h₂ − 0.5)). AND and OR share the same weights. Only their biases differ, which shows how much work the bias does. The output computes OR but not AND. One hidden layer of two neurons is also enough for XOR.

What is the bias term in a neural network?

The bias is a learned number added to a neuron's weighted sum before the activation function, so the neuron computes σ(w·x + b). The weights set which direction the neuron responds to, and the bias sets where along that direction it switches on. Geometrically, the bias moves the neuron's decision boundary away from the origin.

Nearly every dense and convolutional layer has one bias per neuron, so a layer with n inputs and m neurons holds n×m weights and m biases. Without biases, every boundary in the first layer would have to pass through the origin, which fails as soon as the classes sit off-center. This explainer starts with one neuron and ends with a small network whose boundaries combine to solve XOR.

Key ideas

  • A neuron computes σ(w·x + b); the bias b is learned just like the weights.
  • The weights set the decision boundary's orientation, and the bias sets its offset.
  • Without a bias, every boundary must pass through the origin.
  • Each neuron draws one linear boundary; later layers combine them into curved regions.
  • XOR needs at least two boundaries, so it needs at least one hidden layer.
  • Biases add few parameters but let each neuron place its boundary anywhere.

Common questions

Do all neural network layers have a bias?

No. A layer followed by batch normalization usually drops its bias, because batch norm subtracts the mean and then adds its own learned shift. Some large language models, such as PaLM, remove biases from their dense layers entirely and report more stable training.

Is the bias term related to bias in the bias-variance tradeoff?

No, the shared word is a coincidence. The bias term is a parameter inside a neuron. Bias in the bias-variance tradeoff is a model's systematic error, caused by assumptions too simple to fit the data.

Can a decision boundary be curved?

A single neuron's boundary is always flat: a line, plane, or hyperplane. Curved boundaries come from combining many neurons in hidden layers, from adding non-linear features such as x₁² before a linear model, or from kernel methods such as an RBF support vector machine.

How is a decision boundary plotted for a trained model?

Evaluate the model on a fine grid of points covering the input space and color each point by its predicted class, as the shaded regions here do. The boundary is where the color changes. This only works directly for two input features; for more, plots fix the other features or project the data to 2D first.

Related explainers

References

  1. Goodfellow, Bengio & Courville (2016), "Deep Learning", chapter 6: Deep Feedforward Networks (6.1: Learning XOR)
  2. Nielsen, "Neural Networks and Deep Learning", chapter 1
  3. Ioffe & Szegedy (2015), "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift"
  4. Chowdhery et al. (2022), "PaLM: Scaling Language Modeling with Pathways"

Last updated