You are given a pretrained image classification network, such as ResNet. How would you adapt it to perform video classification, ensuring that both spatial and temporal information are captured?
Please discuss possible architectural modifications and trade-offs between different approaches.
Answer
Adding temporal modeling is essential when adapting an image-classification CNN for video classification. Options include:
(1) 3D CNNs (C3D/I3D):
Extend 2D convs to 3D to learn motion. Inflating 2D convolutions into 3D. (Example: I3D inflates 2D ResNet filters into 3D filters using pretrained ImageNet weights.)
Pros: Superior capability to capture fine-grained motion and spatio-temporal features.
Cons: High computational cost and high demand for video training data (Can be mitigated by I3D-style inflation).
(2) Combining frame-level CNN features with RNN/LSTM/TCN/Transformer:
Use CNN for spatial feature extraction, sequence model for temporal modeling with the extracted spatial features.
Pre-train CNN on the target dataset to conduct image classification might improve the performance.
Pros: Leverages powerful 2D CNN pre-training easily, lower computational cost. Flexible, handles variable-length sequences and can be good at modeling long-term sequence dependencies.
Cons: Less effective at modeling local and subtle motion features without further specialize temporal modeling design.
The below figure demonstrate the CNN combined with RNN/LSTM/TCN/Transformer modelling process.
(3) Temporal pooling/attention:
Simple frame aggregation with average/max pooling or attention.
Pros: Lightweight, efficient. Useful when frame order is less critical or resources are limited.
Cons: May lose fine-grained motion cues.
Leave a Reply