Interview for Machine Learning

Category: Medium

  • DL0003 1×1 Convolution

    What are the benefits of using 1×1 convolutional layers in deep learning architectures?

    Answer

    A 1×1 convolution, also known as a pointwise convolution, is a convolutional operation where the kernel size is 1×1, which plays several crucial roles in deep learning architectures.
    (1) Dimensionality control: 1×1 convolution can reduce or expand the number of feature maps, trading off representational capacity and computational cost.

    For example, Bottleneck designs: In architectures like ResNet’s bottleneck block, a 1×1 conv first reduces channels (e.g., 256→64), then a 3×3 conv processes those, and finally another 1×1 conv expands back (64→256) to restore capacity while keeping compute manageable.

    (2) Increased Network Depth with Controlled Cost: Allows for the design of deeper networks by reducing channel dimensionality before computationally expensive spatial convolutions.
    (3) Cross-Channel Feature Fusion: Enables interaction and combination of information across different feature channels at the same spatial location.
    (4) Non-linear mixing: When followed by activations (ReLU, etc.), 1×1 convolutions introduce non‐linear channel mixing that enhances model expressiveness.

    Login to view more content

  • DL0002 All Ones Init

    What are the potential consequences of initializing all weights to one in a deep learning model?

    Answer

    Below are the key consequences of initializing all weights in a deep-learning model to one (a constant non-zero value), illustrating why random, scaled initializations (e.g., Xavier/He) are essential.
    (1) Symmetry Problem: Neurons receive identical gradients, causing them to learn the same features rather than developing distinct representations.
    (2) Limited Representational Capacity: The network cannot capture complex, varied patterns because all neurons behave identically.
    (3) Slow/No Convergence: The lack of Representational Capacity further makes it difficult for the model to update to the optimal weights. (The below image shows an example for training loss comparison for ones initialization vs random initialization)

    (4) Activation Saturation: Can push neurons into saturated regions of activation functions (e.g., sigmoid, tanh), leading to vanishing gradients.

    Login to view more content

  • DL0001 Residual Connection

    Why are residual connections important in deep neural networks?

    Answer

    Residual connections, also known as skip connections, are vital in deep neural networks primarily because they tackle the infamous vanishing gradient problem and help with the related issue of network degradation as the network depth increases.

    Residual connection is often expressed by the following equation:
    y = F(x) + x
    Where:
    F(x) represents the residual mapping that the network learns (i.e., what needs to be added to the input x to achieve the desired output).
    x is the input to the residual block.

    (1) Tackle vanishing gradient problem:
    Residual connections create a direct shortcut for gradient flow by incorporating an identity mapping into the learned transformation. This ensures that even if the gradient through the learned component is small, a strong, direct gradient component persists, preventing vanishing gradients in deep networks. This improves gradient flow during backpropagation, reducing vanishing gradients and enabling the training of very deep networks.

    (2) Address network degradation:
    Residual connections mitigate the degradation problem often seen in deep networks. Without these connections, simply stacking more layers can result in higher training errors, as the network struggles to update its weights effectively. With residual connections, any layer that doesn’t contribute useful information can effectively learn to output zeros in the residual branch, letting the network default to an identity mapping.

    Login to view more content