DL0117 3D Attention vs Frame Pooling

Both families turn frames of patch tokens into one clip representation, and they differ only in where temporal information is allowed to mix. Frame-level pooling runs a purely spatial encoder per frame and then averages or attention-pools the frame vectors (frames → per-frame ViT → pool → head), so no token ever sees another frame and the aggregation is permutation invariant over time. 3D modules mix earlier: a 3D convolution gives each token a local spatiotemporal receptive field of size , joint space-time attention lets all tokens attend to each other, and TimeSformer’s divided space-time attention factorizes that into a temporal MSA over the same spatial position across frames followed by a spatial MSA inside each frame. The cost separation is the first thing to state in an interview: joint attention is in the pair count while divided is , a ratio of that reaches roughly 64x at 96 frames. The accuracy separation depends almost entirely on whether the label actually depends on frame order: on scene-biased Kinetics-400 dropping temporal attention costs about a point, while on Something-Something V2 the same ablation costs roughly 23 points.

(1) Pooling Is Order-Blind By Construction: mean or max pooling over frame embeddings is symmetric, so “opening a door” and “closing a door” produce the identical clip vector no matter how good the image backbone is.
(2) 3D Convolution Buys Locality Cheaply: cost is linear in , but the temporal receptive field grows only frames per layer, so long-range order needs depth or a slow/fast dual pathway.
(3) Joint Attention Is Global But Quadratic: with ViT-B at 8 frames the token sequence is , and every added frame inflates the attention matrix quadratically.
(4) Factorization Is The Practical Default: divided space-time attention beat both space-only and joint attention in the TimeSformer ablations while being far cheaper than joint, which is why factorized variants dominate video ViTs.
(5) Benchmark Bias Decides The Verdict: appearance-biased datasets reward a strong image backbone, and temporally-ordered datasets punish any aggregator that discards order.
(6) Pooling Keeps System Properties Attention Destroys: per-frame embeddings can be cached, indexed, and streamed independently, which is why large-scale video retrieval still ships CLIP-style pooled encoders.

Figure 1: The three aggregation schemes differ only in the attended set of a query token. Space-only attention plus pooling never crosses a frame boundary, divided space-time adds a one-dimensional temporal pass over the same spatial position, and joint attention connects all tokens at quadratic cost.

A subtlety that separates mid from senior candidates is that divided attention is not simply “cheaper joint attention”. Its temporal MSA only compares a patch with the same spatial coordinate in other frames, so a fast-moving object that shifts several patches between frames is matched indirectly, through the spatial MSA that follows. That works because the two passes alternate at every block, but it is also why divided attention benefits from higher frame rates and larger patch strides, and why 3D convolutions with a spatial kernel remain competitive on motion-heavy, short-horizon tasks. Practically, inflating an image-pretrained ViT into a divided model requires zero-initializing the temporal projection so the network starts as an exact image model and the pretrained features survive the first epochs.

Mathematical Formulation:

Where:

• is the clip embedding and the frozen or fine-tuned per-frame encoder applied to frame ; because the sum is symmetric, is unchanged by any permutation of the frames.
• indexes frames, is the number of patch tokens per frame ( for ViT-B at with patch 16), and is the model width.
• , , and are the attention costs of space-only, joint space-time, and divided space-time blocks; all three exclude the identical per-token MLP term.
• and are the temporal and spatial kernel sizes of a 3D convolution, whose cost is linear in but whose temporal receptive field after layers is only about frames.
• The ratio holds whenever is at least 2; it grows toward as grows, so the saving is bounded above by the token count per frame.

Figure 2: TimeSformer ViT-B ablations at 8 frames. Removing temporal mixing costs only 1.1 points on Kinetics-400, whose classes are largely identifiable from a single frame, but 22.9 points on Something-Something V2, where the label is defined by the direction of motion.