Interview for Machine Learning

Author: admin

  • DL0023 Dilated Convolution

    What are dilated convolutions? When would you use them?

    Answer

    Dilated convolutions enhance standard convolution by inserting gaps between filter elements, thereby allowing the network to gather more context (a larger receptive field) without an increase in parameters or a reduction in resolution.

    Dilated convolutions (also known as atrous convolutions) modify standard convolution by inserting gaps (zeros) between kernel elements. A “dilation rate” dictates the spacing of these gaps. A dilation rate of 1 is a standard convolution.

    Contrast with Pooling:
    Pooling reduces spatial resolution (downsamples) while increasing the receptive field.
    Dilated convolutions increase the receptive field without reducing resolution.

    Multi-Scale Feature Extraction:
    By adjusting the dilation rate, these convolutions can aggregate features from both local neighborhoods and larger regions, making it easier for the network to learn from multi-scale context.

    Common Use Cases: Any task needing large receptive fields without downsampling.
    (1) Semantic segmentation (e.g., DeepLab): Expand the receptive field and capture multi-scale context.
    (2) Audio processing (e.g., WaveNet): Model long-range temporal dependencies.

    Here is a 1D Dilated Convolution illustration.

    Here is a 2D Dilated Convolution illustration.

    Login to view more content

  • DL0022 CNN Architecture

    Describe the typical architecture of a CNN.

    Answer

    A Convolutional Neural Network (CNN) is structured to efficiently recognize complex patterns in data. It begins with an input layer that feeds in raw data. Convolutional layers then extract key features using filters, which are enhanced through non-linear activation functions like ReLU. Pooling layers are used to reduce the size or dimensions of these features, thereby improving computational efficiency and promoting invariance to small shifts. The extracted features are flattened and passed through fully connected layers that culminate in an output layer for final predictions, typically employing a softmax function for classification tasks. Optional techniques, such as dropout and batch normalization, further refine learning and help prevent overfitting.

    (1) Input Layer: Accepts raw data as multi-dimensional arrays.
    (2) Convolutional Layers: Use learnable filters (kernels) to scan the input and extract local features.
    (3) Activation Functions: Apply non-linearity (commonly ReLU) after each convolution operation.
    (4) Pooling Layers: Downsample feature maps using techniques like max or average pooling to reduce spatial dimensions and computations.
    (5) Stacked Convolutional and Pooling Blocks: Multiple iterations to progressively extract intricate hierarchical features.
    (6) Flattening: Converts feature maps into one-dimensional vectors.
    (7) Fully Connected Layers: Learn complex patterns and perform decision-making.
    (8) Output Layer: Produces final predictions using appropriate activation functions (e.g., softmax for classification)
    (9) Additional Components (Optional): Dropout for regularization, batch normalization for training stability, and skip connections in more advanced models.

    Below is a visual representation of a typical CNN architecture. Padding is used in convolution to maintain dimensions.

    Login to view more content

  • DL0021 Feature Map

    What is the feature map in Convolutional Neural Networks?

    Answer

    A feature map is the output of a convolution operation in a Convolutional Neural Network (CNN) that highlights where specific features appear in the input, enabling the network to understand patterns and structures in input data.

    Feature Map in CNNs:
    (1) Output of a Filter: It’s the 2D (or 3D) output generated when a single convolutional filter slides across the input data.
    (2) Highlighting a Specific Feature: Each feature map represents the spatial locations and strengths where a particular pattern or characteristic (e.g., a vertical edge, a specific texture, a corner) is detected in the input.
    (3) Multiple Feature Maps per Layer: A convolutional layer typically uses multiple filters, with each filter producing its unique feature map.

    The following example shows feature map examples calculated with different filters on the original image.

    Login to view more content

  • DL0020 CNN Parameter Sharing

    How do Convolutional Neural Networks achieve parameter sharing? Why is it beneficial?

    Answer

    Convolutional Neural Networks (CNNs) share parameters by using the same convolutional filter across different spatial locations, enabling them to learn location-independent features efficiently with fewer parameters and better generalization.

    How CNNs Achieve Parameter Sharing:
    (1) Convolutional Filters/Kernels: A small matrix of learnable weights (the filter) is defined.
    (2) Sliding Window Operation: This filter slides across the entire input image (or feature map).
    (3) Weight Reuse: The same weights within that filter are used to compute outputs at every spatial location where the filter is applied.

    Why Parameter Sharing is Beneficial:
    (1) Reduced Parameters: Significantly fewer learnable parameters compared to fully connected networks.
    (2) Translation equivariance: Detects features regardless of their position in the image.
    The following example demonstrates translation equivariance using a CNN-like convolution with a shared filter.

    (3) Improved Generalization: Less prone to overfitting due to fewer parameters.
    (4) Computational Efficiency: Faster training and inference.

    Login to view more content

  • DL0019 Go Deep

    How does increasing network depth impact the learning process?

    Answer

    Increasing network depth enhances feature learning and model power, but brings training instability, higher cost, and design complexity.

    Increasing network depth can bring benefits:
    (1) Improved Feature Hierarchy: Deeper layers can learn more abstract, high-level features. In image classification, early layers learn edges, deeper ones learn shapes and objects.
    (2) Increased Model Capacity: More layers allow the network to model more complex functions and patterns.
    (3) Improved Efficiency for Complex Functions: For certain complex functions, deep networks can represent them more efficiently with fewer neurons compared to shallow ones.

    Increasing network depth can bring challenges:
    (1) Vanishing/Exploding Gradients: Gradients can become extremely small or large as they propagate through many layers, hindering effective training, e.g., “Without techniques like skip connections, a 100-layer network might struggle to learn because gradients vanish before reaching early layers.“
    (2) Increased Computational Cost (Challenge): Training deeper networks requires significantly more computational resources and time.
    (3) Higher Data Requirements (Challenge): Deeper models have more parameters and are more prone to overfitting if not trained on large datasets.

    The following example visually compares a shallow and a deep neural network on learning a complex function.

    Login to view more content

  • DL0018 NaN Values

    What are the common causes for a deep learning model to output NaN values?

    Answer

    NaN outputs in deep learning usually stem from unstable math operations, gradient issues, bad hyperparameters, or data problems. Prevent this with proper initialization, proper normalization, stable activation functions, and well-tuned hyperparameters.

    Here are the common causes for a deep learning model to output NaN values:
    (1) Exploding Gradients: Gradients become excessively large during training, leading to NaN weight updates
    (2) Numerical Instability: Operations like log(0), division by zero, or square roots of negative numbers. Without a small constant (epsilon) in its denominator, batch normalization will suffer from division by zero if a batch has zero variance.
    (3) Improper Learning Rate: Too high a learning rate can cause parameter updates to diverge and push model parameters to extreme values.
    (4) Incorrect Weight Initialization: Incorrectly initializing all weights to very large positive numbers can cause activations to overflow immediately.
    (5) Data Issues: Input data contains NaN or extreme values.

    Login to view more content

  • DL0017 Reproducibility

    How to ensure the reproducibility of the deep learning experiments?

    Answer

    Reproducibility in deep learning is achieved by controlling randomness via fixed seeds and deterministic operations, maintaining strict code and dependency versioning, managing datasets carefully, and keeping comprehensive logs of all experiment settings. These practices ensure that experiments can be reliably repeated and validated, regardless of external factors.

    (1) Seed Control and Deterministic Operations:
    Set random seeds for all libraries (Python, NumPy, TensorFlow/PyTorch).
    Enable deterministic settings in your deep learning framework to reduce nondeterminism.
    (2) Code Versioning and Configuration Management:
    Use version control systems like Git.
    Maintain detailed configuration files (using YAML or JSON) that log hyperparameters and settings for each experiment.
    (3) Environment and Dependency Control:
    Use virtual environments (e.g., Conda) or containerize your projects with Docker.
    Freeze library versions to ensure consistency in the software environment.
    (4) Dataset Management:
    Fix train-test splits and document data preprocessing steps.
    Use versioned or static datasets to prevent unintentional changes over time.
    (5) Logging and Documentation:
    Log hardware details, random seeds, and experiment configurations.
    Utilize experiment tracking tools (like MLflow or Weights & Biases) to archive training runs and parameters.

    Below is one example that illustrates the experiments are not reproducible.

    Login to view more content

  • ML0047 Parameters

    What are the differences between parameters and hyperparameters?

    Answer

    Parameters are the values that a model learns from its training data, while hyperparameters are settings defined by the user that guide the training process and model architecture.

    Parameters:
    (1) Internal variables learned from data (e.g., weights and biases).
    (2) Adjusted during training using optimization algorithms.
    (3) Capture the model’s learned patterns and information.

    Hyperparameters:
    (1) External configurations set before training (e.g., learning rate, batch size, number of layers).
    (2) Remain fixed during training and are not updated by the learning process.
    (3) Influence how the model learns and its overall structure.

    Login to view more content

  • DL0016 Learning Rate Warmup

    What is Learning Rate Warmup? What is the purpose of using Learning Rate Warmup?

    Answer

    Learning Rate Warmup is a training technique where the learning rate starts from a small value and gradually increases to a target (base) learning rate over the first few steps or epochs of training.

    Purpose of Using Learning Rate Warmup:
    (1) Stabilizes Early Training: At the beginning of training, weights are randomly initialized, making the model sensitive to large updates. A warmup gradually increases the learning rate, preventing unstable behavior.
    (2) Allow Optimizers to Adapt: Optimizers like Adam and AdamW rely on gradient statistics that can be unstable at the start. Warmup allows these optimizers to accumulate more accurate estimates before using a high learning rate.
    (3) Enables Large Batch Training: Mitigates issues that can arise when combining a large batch size with a high initial learning rate.

    Below shows an example using Learning Warmup followed by Cosine Decay.

    Login to view more content

  • ML0046 Forward Propagation

    Please explain the process of Forward Propagation.

    Answer

    Forward propagation is when a neural network takes an input and generates a prediction. It involves systematically passing the input data through each layer of the network. A weighted sum of the inputs from the previous layer is calculated at each neuron, and then a nonlinear activation function is applied. This process is repeated layer by layer until the data reaches the output layer, where the final prediction is generated.

    Here is the process of Forward Propagation:
    (1) Input Layer: The network receives the raw input data.
    (2) Layer-wise Processing:
    Linear Combination: Each neuron calculates a weighted sum of its inputs and adds a bias.
    Non-linear Activation: The resulting value is passed through an activation function (e.g., ReLU, sigmoid, tanh) to introduce non-linearity.
    (3) Propagation Through Layers: The output from one layer becomes the input for the next layer, progressing through all hidden layers.
    (4) Output Generation: The final layer applies a function (like softmax for classification or a linear function for regression) to produce the network’s prediction.

    Login to view more content