You are given a pretrained image classification network, such as ResNet. How would you adapt it to perform video classification, ensuring that both spatial and temporal information are captured?
Please discuss possible architectural modifications and trade-offs between different approaches.
The Problem: a pretrained 2D CNN understands each frame in isolation. Fill the question mark with an adaptation that adds temporal modeling without throwing away what the image network already knows.
Answer
The missing ingredient is temporal modeling: a 2D CNN sees each frame independently, so any adaptation must add a mechanism that aggregates information across time. Three families cover the design space: inflate the network into 3D convolutions, keep the 2D CNN as a per-frame encoder and add a sequence model on top, or simply pool or attend over per-frame features. The right choice trades temporal modeling power against compute cost and data hunger.
(1) 3D CNNs (C3D / I3D): Extend 2D convolutions into 3D to learn motion directly; I3D inflates pretrained 2D filters into 3D, keeping ImageNet weights useful.
(2) CNN + Sequence Model: Use the 2D CNN as a per-frame feature extractor, then run an LSTM / TCN / Transformer over the feature sequence for temporal modeling.
(3) Temporal Pooling / Attention: Aggregate per-frame features with average/max pooling or attention: cheapest, but frame order and fine motion cues fade.
Figure 1: The CNN + sequence model option: the pretrained 2D CNN is reused per frame, and a temporal model (LSTM/TCN/Transformer) learns how features evolve. This is the balanced default: full ImageNet leverage, moderate cost, natural variable-length handling.
Clarify Before Designing:
(1) Latency: real-time or offline? What is the per-clip inference budget?
(2) Input Statistics: fps, clip duration, resolution: trimmed clips or untrimmed streams?
(3) Data Scale: how much labeled video is available: hundreds or hundreds of thousands of clips?
(4) Compute: can training and inference afford 3D-convolution cost?
(5) Motion Sensitivity: do classes depend on subtle motion (opening vs closing) or mostly on scenes and objects?
Leave a Reply