ML0037 Bias in NN

The bias term in a neuron () is the learnable offset that shifts the activation function’s threshold. Without it, every neuron’s pre-activation would be a strictly linear (origin-passing) function of its inputs: a neuron could only fire proportionally to its input, and every decision boundary or fitted function would be forced through the origin. The bias removes that constraint: it lets a neuron activate even when its weighted input sums to zero, and lets decision boundaries sit anywhere in input space, not just through the origin. This adds crucial flexibility for approximating real-world functions, compensates for systematic offsets in the data, and plays a role loosely analogous to the firing threshold of a biological neuron: the bias sets how much input stimulation is needed before the neuron becomes active. In short, the bias is to a neuron what the intercept is to linear regression: a small parameter with an outsized effect on representational power.

(1) Shifts the Threshold: The bias moves the activation curve left/right, so a neuron can fire (or stay off) at any input level.
(2) Escapes the Origin: Without bias, boundaries and fitted functions are forced through (0,0); with it they can sit anywhere.
(3) Flexibility: One extra learnable parameter per neuron that absorbs systematic offsets and improves approximation.

Figure 1: Why bias matters in one picture: the data’s trend clearly does not pass through the origin. The no-bias model (orange) is constrained through (0,0) and misfits everywhere; the model with a bias term (blue) shifts the line up and fits the trend.

Mathematical Formulation:

Where:

• are the inputs, the weights, the bias, the pre-activation, the output.
• acts as the effective threshold: the third line says the unit crosses its threshold exactly when the weighted input exceeds . With the neuron activates more easily (even at zero input); with a negative bias it requires stronger input to fire.