DL0116 Video VLM Temporal Sampling

A video VLM has no separate temporal-resolution dial: it flattens the stream into one token sequence, so frame rate, tokens per frame, and clip length all collapse into a single token count , and that count fixes both the KV cache size (linear in ) and the prefill cost (quadratic in ). Two hours sampled at 1 fps with 64 tokens per frame is 460,800 tokens; on a Llama-3-8B-class decoder with 32 layers, 8 KV heads of head dimension 128, and a bf16 cache, each token costs 128 KB of KV, so the cache alone is roughly 60 GB, against the roughly 62 GB an 80 GB H100 has free after weights and activations. Multiply the frame rate by four or keep the encoder’s full 256 tokens per frame and you are at 240 GB, which no single device holds. Production systems therefore invert the question: fix a token budget from the memory and latency SLO first, then decide how to spend it across time and space. The three levers are temporal subsampling (fewer frames), token compression (pooling, merging, or resampling each frame), and bounded state (a sliding window plus a compressed memory bank or an offline retrieve-then-read pass), and only the third one makes memory independent of stream length.

(1) Tokens Are The Currency: memory scales as and prefill attention as , so a 4x frame-rate increase is a 4x memory bill and a 16x prefill bill.
(2) Per-Token KV Cost Is Architectural: bytes, so GQA with 8 KV heads instead of 32 already cuts it 4x, and INT4 KV quantization cuts it another 4x before any sampling change.
(3) Equal Memory, Different Failures: halving and quartering can cost the same tokens, but the first causes temporal aliasing of short events while the second destroys small text and fine spatial detail.
(4) Slow-Fast Splitting: strong systems decouple the two axes, keeping many frames at very few tokens for motion and a handful of keyframes at full spatial resolution for detail.
(5) Adaptive Beats Uniform At Fixed Budget: dropping near-duplicate frames by feature similarity and selecting query-relevant segments spends the same tokens on the informative part of the timeline.
(6) Time Grounding Must Survive Downsampling: position encodings need absolute timestamps, not token indices, or a variable frame rate makes the model’s answers about “when” drift.

Figure 1: KV cache for the same two-hour video under six budgets. Memory depends only on the product and the per-token cost, so sampling changes and cache quantization are interchangeable for memory, but only a bounded streaming state stops growth with stream length.

The reason the two sampling axes are not interchangeable for accuracy is that they alias different things. Uniform sampling at 1 fps is enough for plot-level questions over an hour of footage but silently deletes any event shorter than a second, which is why counting, ordering, and “who handed what to whom” questions collapse at low frame rates. Cutting tokens per frame is safe for gist and scene recognition and catastrophic for anything requiring OCR of on-screen text or small-object detail. Public systems make these trade-offs explicit: Gemini samples video at 1 fps with about 258 tokens per frame by default, which is roughly 0.93M tokens per hour and effectively consumes a 1M-token context, and offers a low-resolution mode at about 66 tokens per frame that stretches the same window to several hours. For genuinely unbounded streams the only stable design is bounded state: keep the last few tens of seconds as an exact KV window with attention sinks so the distribution stays in-domain, merge evicted frames into a fixed-size memory bank, and for offline archives replace dense prefill with a two-pass pipeline that indexes cheap summaries and then re-decodes only candidate segments at high frame rate.

Mathematical Formulation:

Where:

• is the total visual token count fed to the decoder, the sampling rate in frames per second, the stream duration in seconds, and the tokens retained per frame after pooling or resampling.
• is the KV bytes per token, with layers, key-value heads, head dimension , and bytes per stored element; the factor 2 counts keys and values.
• is cache size and the free HBM after weights, activations, and the text prompt; is the highest frame rate a dense prefill can afford at a given .
• is prefill attention cost with model width ; it usually binds before memory does, since doubling quadruples time to first token.
• The numeric example uses , , and bytes, giving 128 KB per token, and s at , , so GB.
• A streaming design replaces with , the exact window plus a fixed memory bank, which is in .

Figure 2: A streaming video VLM: decode rate and per-frame token count shrink the sequence by two orders of magnitude, then eviction into a fixed memory bank makes resident state constant in stream length while the exact sliding window preserves fine detail for recent seconds.