Interview for Machine Learning

Category: Easy

  • DL0006 Layer Freeze in TL

    What are the common strategies for layer freezing in transfer learning?

    Answer

    Layer freezing controls which pre-trained weights are updated during fine-tuning. Frozen layers keep their learned representations intact, while trainable layers adapt to the target task. The right choice balances leveraging general features against adapting higher-level representations, and depends mainly on target dataset size and task similarity.

    (1) Freeze All but the Output Layer(s): Train only the final classification/regression layers; a good starting point for similar tasks and small datasets.
    (2) Freeze Early Layers: Early layers capture general features (edges, textures), so train only the later, task-specific layers; effective for moderately similar tasks.
    (3) Fine-Tune All Layers with a Low Learning Rate: Adapt all weights slowly; use with caution on small datasets to avoid catastrophic forgetting.
    (4) Gradual Unfreezing: Start with frozen layers and progressively unfreeze during training, avoiding large early updates that can destroy learned features.
    (5) Backbone-Freeze, Then Low-LR Fine-Tune: Freeze the backbone until the new head converges, then unfreeze it and continue with a reduced learning rate.

    Four layer-freezing strategies on a five-block backbone plus head, showing which blocks are frozen versus trainable for each strategy.

    Figure 1: The four common strategies differ in how many blocks stay frozen; freeze more when data is scarce and tasks are similar.

    Mathematical Formulation:
    \theta = \theta_{frozen} \cup \theta_{trainable}
    \Delta\theta_{frozen} = 0
    \theta_{trainable} \leftarrow \theta_{trainable} - \eta\,\nabla_{\theta_{trainable}}\mathcal{L}

    Where:

    • \theta_{frozen} is the parameter subset kept fixed at its pre-trained values; only \theta_{trainable} receives gradient updates.
    • \eta is the learning rate; strategies (3) and (5) use a reduced \eta (e.g., 0.1×) to protect pre-trained features.
    Step chart showing the percentage of trainable layers increasing from head-only to all layers across training epochs during gradual unfreezing.

    Figure 2: Gradual unfreezing starts with the head and unfreezes deeper blocks step by step with a reduced learning rate.

    Login to view more content

  • DL0005 Transfer Learning

    Why use transfer learning in deep learning instead of training from scratch?

    Answer

    Transfer learning reuses knowledge from a pre-trained model to improve performance, reduce training time and data requirements, and lower computational cost on a new but related task. Instead of learning low-level features from random weights, the model starts from representations that already capture generalizable patterns such as edges, textures, and shapes, so far less target data is needed to reach strong accuracy.

    (1) Leverages Existing Knowledge & Reduced Data Requirements: Pre-trained weights encode useful representations learned from large datasets, so good performance is possible with significantly less task-specific data.
    (2) Faster Convergence & Training Time: Starting from pre-trained weights is a much better initialization than random weights, leading to faster convergence and often better local optima.
    (3) Improved Performance on Limited-Data Tasks: When data is scarce, transfer learning typically yields higher accuracy and better generalization than training from scratch.

    Transfer learning pipeline showing a backbone pre-trained on a large source dataset being copied to a target task where the backbone is frozen or fine-tuned with a low learning rate and a new head is trained.

    Figure 1: The backbone’s weights are copied from pre-training; only the new head (and optionally upper layers) must be learned from limited target data.

    Mathematical Formulation:
    \theta^* = \arg\min_{\theta}\ \mathcal{L}_{target}(\theta;\ \theta_{init} = \theta_{pretrained})
    \eta_l = \eta_{base} \cdot \gamma^{\,L-l}

    Where:

    • \theta_{pretrained} is the weight set learned on the source task; fine-tuning minimizes the target loss starting from this initialization.
    • \eta_l is the learning rate of layer l out of L; a decay factor \gamma below 1 gives earlier (more general) layers smaller updates than later (task-specific) layers.
    Log-scale chart showing transfer learning achieving high validation accuracy with little target data while training from scratch needs much more data to catch up.

    Figure 2: Transfer learning dominates in the low-data regime; the advantage shrinks as the target dataset grows.

    Faster Convergence: Because pre-trained weights are already a strong initialization, the model reaches its accuracy plateau in far fewer epochs than training from random weights.

    Accuracy versus epoch chart showing transfer learning converging quickly to a high plateau while training from scratch improves slowly over many more epochs.

    Figure 3: Transfer learning converges faster and plateaus higher; training from scratch improves slowly over many epochs.

    Login to view more content

  • DL0004 Small Kernels

    What are the key advantages of using small convolutional kernels, such as 3×3, over utilizing a few larger kernels in deep learning architectures?

    Answer

    Using small convolutional kernels instead of a few larger kernels lets a network reach the same receptive field with fewer parameters, more depth, and more non-linearity. A stack of 3\times3 convolutions matches the spatial coverage of a single large kernel while inserting an activation between each layer, which increases the network’s representational power. This design is the foundation of architectures such as VGG.

    (1) Deeper Networks & More Non-Linearity: Stacking multiple 3\times3 layers (e.g., three of them) creates a deeper network with more non-linear activation functions than a single large kernel.
    (2) Reduced Parameters: Multiple small kernels achieve the same receptive field as a larger one with fewer parameters: two stacked 3\times3 layers use 18\cdot C_{in}\cdot C_{out} weights versus 25\cdot C_{in}\cdot C_{out} for one 5\times5 layer.
    (3) Computational Efficiency: Fewer parameters generally mean lower FLOPs during training and inference.
    (4) Gradual Receptive Field Expansion: Successive 3\times3 convolutions progressively build a larger receptive field while preserving local detail capture, ideal for textures and edges.

    Two stacked 3x3 convolutions viewing the same 5x5 input region as a single 5x5 convolution.

    Figure 1: Two stacked 3\times3 convolutions see the same 5\times5 input region as one 5\times5 convolution, but with an extra non-linearity in between.

    Kernel Composition: Two stacked 3\times3 kernels compose into a single equivalent 5\times5 kernel (their full discrete convolution), which is exactly why the receptive fields match.

    Two all-ones 3x3 kernels composing into an equivalent 5x5 kernel with coefficients formed by their full discrete convolution.

    Figure 2: Two 3\times3 kernels compose into one equivalent 5\times5 kernel; stacking keeps them separate: 18 weights plus an extra non-linearity instead of a 25-weight merge.

    Mathematical Formulation:
    P_k = k^2 \cdot C_{in} \cdot C_{out}
    RF_l = 1 + l\cdot(k-1)
    2 \times 3^2 = 18 \text{ vs } 5^2 = 25
    3 \times 3^2 = 27 \text{ vs } 7^2 = 49

    Where:

    • P_k is the parameter count of one k\times k convolution (ignoring biases).
    • C_{in} and C_{out} are the input and output channel counts.
    • RF_l is the receptive field after l stacked layers; two 3\times3 layers give RF=5, three give RF=7.
    Bar chart comparing parameter counts of 5x5 and 7x7 convolutions against stacked 3x3 convolutions with equal receptive fields.

    Figure 3: For an equal receptive field, stacked 3\times3 convolutions cut parameters by 28% (vs 5\times5) and 45% (vs 7\times7).

    Login to view more content