DL0167 Video Generation Physical Consistency Evaluation

FID and FVD compare summary statistics of clip embeddings from appearance-biased backbones (InceptionV3, I3D), so they answer whether a set of generated videos resembles a reference set, never whether one clip obeys physics. They are dominated by per-frame content rather than motion, they are nearly insensitive to short-range frame reordering, and they can improve while dynamics get worse. Physical consistency therefore needs per-clip, falsifiable probes. The three that work in practice are: condition the model on the opening frames of a real recorded event and score the predicted continuation against the actual future; ask a calibrated judge (human, or a VLM validated against human labels) for a binary physical-commonsense verdict on each clip; and extract a symbolic readout (track → lift to 3D → fit dynamics) so gravity, momentum, permanence, and flow coherence become numbers with known correct values. For a world model you add a fourth layer, closed-loop rollout, because the only decision-relevant question is whether a policy trained or planned inside the model still succeeds in the real environment.

(1) Distributional vs Per-Clip: FVD is a set-level Frechet distance and cannot label a single video as physically wrong, so it belongs in a regression suite, not in a physics report.
(2) Real-Continuation Probes: give the model the first frames of a genuine recorded event and compare the continuation to ground truth with spatial IoU of the motion mask and masked pixel MSE.
(3) Calibrated Judges: report physical commonsense (PC) and semantic adherence (SA) separately plus their joint pass rate, and publish the judge’s agreement with humans before trusting it.
(4) Symbolic Readout: tracking plus depth lifting turns physics into scalars: fitted gravity residual, momentum error at collisions, occlusion recovery rate, and optical-flow warp error.
(5) Object Permanence Is A Timeline Test: it is only measurable across an occlusion window, so probes must be authored with an occluder and scored on identity, count, and attribute preservation after reappearance.
(6) Closed-Loop For World Models: action-conditioned rollouts are scored by drift over horizon and downstream task success, which is the metric that actually correlates with usefulness.

Figure 1: The evaluation stack. Only tiers 2 to 4 return a verdict for an individual clip; tier 1 is a distribution-level sanity check whose score can fall (improve) while gravity and collision behaviour degrade, which is exactly why it is drawn with a dashed contribution.

Each of the four physical axes needs a different probe design. Gravity is measured by tracking a free-falling centroid, fitting a quadratic to its height, and comparing the implied acceleration to the reference: generated video overwhelmingly fails toward floaty, slow-motion dynamics with a fitted value well below the true one. Collision is measured at the impact frame, where you check momentum and energy: a coefficient of restitution above 1, interpenetration, or objects passing through each other are hard violations that no appearance metric sees. Object permanence requires a prompt authored with an occluder, then a check that the object reappears with the same identity, count, and colour; the natural scores are the recovery rate and identity-switch count borrowed from multi-object tracking. Temporal coherence is the cheapest to automate: warp consecutive frames with optical flow, mask out occlusions and disocclusions, and average the residual, which catches flicker, texture swimming, and morphing that a per-frame FID is blind to. Crucially, all four scores are reported as per-clip pass rates over a curated probe set, not as an average of a continuous score, because averaging hides the fact that most failures are categorical.

Mathematical Formulation:

Where:

• is the FVD value, with and the mean and covariance of real and generated clip features from a fixed video backbone; it is defined only over sets, which is the root of its uselessness for per-clip physics.
• is the tracked height of an object at time , and the fitted initial height and velocity, and the least-squares acceleration recovered from the generated clip; is its relative error against the reference .
• and are total momentum immediately before and after a contact frame, so is the relative momentum violation; the same construction on kinetic energy exposes a restitution above 1.
• is the number of authored occlusion events and the number where the object reappears with correct identity, count, and attributes, giving the permanence recovery rate .
• is backward warping by the estimated flow , the binary validity mask that removes occluded and out-of-frame pixels, elementwise product, and the averaged warp error over the frames.
• are the physical-commonsense and semantic-adherence verdicts on clip , so is the joint pass rate: a clip counts only if it is both on-prompt and physically plausible.

Figure 2: The symbolic readout on one probe clip. The fitted acceleration is 4.2 instead of 9.8, the object interpenetrates the floor, and the post-impact speed exceeds the pre-impact speed (restitution above 1, so energy is created). Every one of these is a numeric, falsifiable failure; all of them are invisible to a distributional metric computed on the same clip.