ML0007 Dropout

Dropout is a regularization technique used during neural network training to prevent overfitting. At each training step, a fraction of neurons (and their connections) are randomly “dropped out”, meaning their activations are set to zero. This forces the network to learn more robust features, because it cannot rely on any single neuron; instead it learns distributed representations, effectively training an ensemble of many smaller sub-networks that share weights. At inference time every neuron is active, so the model uses all learned features without randomness. To bridge the train/inference gap, inverted dropout scales the active neurons by during training so no adjustment is needed at inference; the alternative standard dropout scales the weights by at inference instead.

(1) Mechanism: Randomly zero a fraction of activations per training step; the dropped set changes every step.
(2) Why It Works: Prevents co-adaptation of neurons and approximates training an exponentially large ensemble of sub-networks.
(3) Training vs Inference: Inference uses the full network; activations are rescaled (during training with inverted dropout, or at inference with standard dropout) so expected magnitudes match.

Figure 1: Left: the full network. Right: one training step with dropout: grayed, crossed-out neurons are zeroed, forcing the remaining sub-network to carry the prediction.

Mathematical Formulation:

Where:

• is the scaled activation of neuron actually passed to the next layer during training.
• is the original activation of neuron , and indexes the neurons in a layer.
• is the binary dropout mask: 1 keeps the neuron, 0 drops it.
• is the drop probability; is the keep probability, and dividing by it is the inverted-dropout scaling that keeps .

Figure 2: Choosing the drop probability : too little dropout leaves overfitting, too much underfits; the test-error minimum is the sweet spot (typically around 0.2–0.5).