DL0005 Transfer Learning

Transfer learning reuses knowledge from a pre-trained model to improve performance, reduce training time and data requirements, and lower computational cost on a new but related task. Instead of learning low-level features from random weights, the model starts from representations that already capture generalizable patterns such as edges, textures, and shapes, so far less target data is needed to reach strong accuracy.

(1) Leverages Existing Knowledge & Reduced Data Requirements: Pre-trained weights encode useful representations learned from large datasets, so good performance is possible with significantly less task-specific data.
(2) Faster Convergence & Training Time: Starting from pre-trained weights is a much better initialization than random weights, leading to faster convergence and often better local optima.
(3) Improved Performance on Limited-Data Tasks: When data is scarce, transfer learning typically yields higher accuracy and better generalization than training from scratch.

Figure 1: The backbone’s weights are copied from pre-training; only the new head (and optionally upper layers) must be learned from limited target data.

Mathematical Formulation:

Where:

• is the weight set learned on the source task; fine-tuning minimizes the target loss starting from this initialization.
• is the learning rate of layer out of ; a decay factor below 1 gives earlier (more general) layers smaller updates than later (task-specific) layers.

Figure 2: Transfer learning dominates in the low-data regime; the advantage shrinks as the target dataset grows.

Faster Convergence: Because pre-trained weights are already a strong initialization, the model reaches its accuracy plateau in far fewer epochs than training from random weights.

Figure 3: Transfer learning converges faster and plateaus higher; training from scratch improves slowly over many epochs.