DL0166 3D Gaussian and NeRF Neural Simulator

A differentiable neural simulator needs one scene description that works twice: as geometry the solver can push around, and as an image the photometric loss can compare against real video. NeRF supplies the second half well, storing the scene as an implicit field that maps a position and view direction to density and radiance, rendered by differentiable volume quadrature along rays. What it lacks is handles, because an MLP field has no particle to give mass, no covariance to advect, and no obvious place to attach a constitutive model. 3D Gaussian Splatting removes that obstacle by storing the scene as an explicit set of anisotropic Gaussians with mean, covariance, opacity, and spherical-harmonic color, so every primitive doubles as a material point in a Lagrangian solver. The simulator advances those points with differentiable physics, usually MLS-MPM, the rasterizer splats them back to pixels, and gradients from the rendering loss flow through both the rollout and the renderer into material parameters such as Young’s modulus, Poisson ratio, yield stress, and friction. The loop is therefore video → physical parameters → rollout of unseen dynamics, with pixels as the only supervision.

(1) Dense Means Physics Everywhere: unlike a mesh, both representations describe the interior volume, so mass, velocity, and stress are defined at every sampled point rather than only on a surface shell.
(2) Gaussians Are Already Particles: a splat’s mean is a particle position and its covariance is a local volume element, which is exactly the state an MPM particle carries, so the reconstruction and the simulation share one data structure.
(3) NeRF Needs A Bridge: a continuous field has to be voxelized or resampled onto particles each step, the Eulerian-Lagrangian conversion that PAC-NeRF introduced, which costs an extra transfer and an extra source of gradient noise.
(4) Covariance Follows The Deformation Gradient: kinematics are preserved by advecting the mean through the flow map and conjugating the covariance with , otherwise stretched material still renders as undeformed blobs.
(5) The Loss Is Purely Photometric: no ground-truth stiffness is available, so system identification is driven by rendered-versus-observed pixel error over a multi-frame rollout.
(6) The Cost Lives In The Backward Pass: memory grows as in substeps and particles, and gradients through contact and friction are non-smooth, so checkpointing and softened contact models are mandatory.

Figure 1: The identification loop. The dense representation appears twice in the graph, once as the initial physical state and once as the thing being rendered, which is why gradients from a single photometric loss can reach material parameters that were never observed directly.

The reason 3D Gaussians displaced radiance fields in this role is mechanical rather than aesthetic. In a Lagrangian solver the unknowns are particle positions and their local deformation, and a Gaussian already stores a mean plus a covariance factored as scale and rotation, so PhysGaussian can drive splats directly with a continuum solver and keep rendering in real time. A radiance field, by contrast, is a function of space with no identity attached to any location, so tracking material requires either a learned deformation field mapping observation time back to a canonical frame, or a particle proxy that carries the physics while the field carries appearance. Two consequences matter in practice. First, the spherical-harmonic coefficients are expressed in world coordinates, so when a splat rotates by its appearance basis must be rotated too, otherwise highlights stay welded to the world frame while the geometry turns. Second, lighting is baked, so shadows and specular reflections do not respond to motion, which biases the photometric gradient exactly on the frames where an object moves most.

Figure 2: Kinematics of a splat. The mean rides the flow map while the covariance is conjugated by the deformation gradient, so shear and compression change both where a Gaussian sits and how it is shaped; skipping the covariance update leaves visibly isotropic blobs inside sheared material and corrupts the rendering gradient.

Mathematical Formulation:

Where:

• is the rendered color of ray , accumulated over ordered samples or splats with per-sample color ; this same alpha-compositing form covers NeRF quadrature and Gaussian splatting.
• is density, the sample spacing along the ray, the resulting opacity, and the transmittance reaching sample .
• is a splat’s covariance, parameterized by rotation and diagonal scale so that it stays positive semi-definite under gradient updates.
• indexes particles, is the flow map from rest to time , and is the deformation gradient whose determinant gives local volume change.
• is the full simulator state (positions, velocities, , affine momentum) and one differentiable substep.
• collects the unknown physical parameters such as Young’s modulus , Poisson ratio , density , yield stress, and friction coefficient.
• is the differentiable renderer and the observed frame, so is a multi-frame photometric residual and its gradient is a product of per-substep Jacobians.

That last product is where the engineering happens. Explicit MPM needs a CFL-limited timestep, so one video frame at 30 fps typically hides a few hundred substeps, and reverse-mode differentiation over a one-second clip means backpropagating through several thousand Jacobians. Storing every intermediate state is what kills the run: 200k particles at roughly 96 bytes of state per particle is about 19 MB per substep, so a 1,000-substep rollout needs 19 GB of tape. The standard remedies are gradient checkpointing at an interval near , truncating the loss window to a handful of frames, and recomputing forward segments during the backward pass. Numerically, the same product of Jacobians makes gradients explode for stiff materials and vanish once the trajectory has been dissipated by friction, which is why practitioners fit a log-parameterized stiffness and soften contact before trusting any gradient at all.

Figure 3: Why differentiable rollouts are short. At 19 MB of particle state per substep, the naive tape exhausts a 40 GB HBM budget after roughly 2,000 substeps, about one second of simulated time, while square-root checkpointing keeps memory in the low gigabytes at the price of one extra forward pass.