DL0006 Layer Freeze in TL

Written by

What are the common strategies for layer freezing in transfer learning?

Answer

Layer freezing controls which pre-trained weights are updated during fine-tuning. Frozen layers keep their learned representations intact, while trainable layers adapt to the target task. The right choice balances leveraging general features against adapting higher-level representations, and depends mainly on target dataset size and task similarity.

(1) Freeze All but the Output Layer(s): Train only the final classification/regression layers; a good starting point for similar tasks and small datasets.
(2) Freeze Early Layers: Early layers capture general features (edges, textures), so train only the later, task-specific layers; effective for moderately similar tasks.
(3) Fine-Tune All Layers with a Low Learning Rate: Adapt all weights slowly; use with caution on small datasets to avoid catastrophic forgetting.
(4) Gradual Unfreezing: Start with frozen layers and progressively unfreeze during training, avoiding large early updates that can destroy learned features.
(5) Backbone-Freeze, Then Low-LR Fine-Tune: Freeze the backbone until the new head converges, then unfreeze it and continue with a reduced learning rate.

Four layer-freezing strategies on a five-block backbone plus head, showing which blocks are frozen versus trainable for each strategy.

Figure 1: The four common strategies differ in how many blocks stay frozen; freeze more when data is scarce and tasks are similar.

Mathematical Formulation:
$\theta = \theta_{frozen} \cup \theta_{trainable}$
$\Delta\theta_{frozen} = 0$
$\theta_{trainable} \leftarrow \theta_{trainable} - \eta\,\nabla_{\theta_{trainable}}\mathcal{L}$

Where:

$\theta_{frozen}$ is the parameter subset kept fixed at its pre-trained values; only $\theta_{trainable}$ receives gradient updates.
$\eta$ is the learning rate; strategies (3) and (5) use a reduced $\eta$ (e.g., 0.1×) to protect pre-trained features.