DL0154 Synthetic Data and Isaac Sim

Nothing in the stack emits a realistic sensor reading directly. Each modality comes out of a different subsystem with its own fidelity ceiling, and realism is added afterwards as an explicit sensor model. RGB and depth are produced by the RTX renderer and read back through Omniverse Replicator annotators, where distance_to_image_plane returns per-pixel metric depth with no holes, no quantization, and no stereo shadows. Joint and contact forces are read out of the PhysX 5 reduced-coordinate articulation solver, so a “measured” wrist force is really a constraint impulse divided by the physics timestep. Tactile is not a first-class sensor at all: it is reconstructed from the signed-distance penetration between a gel collider and the object, then either rendered as a GelSight-style image or collapsed into a taxel normal-force map. Everything the simulator hands you is exact, and the actual engineering is deciding how to break it so a VLA trained on it survives a real RealSense and a real six-axis load cell.

(1) Ground Truth First, Noise Second: the renderer and the solver give error-free signals, so every realism claim rests on a post-processing layer that injects noise, quantization, dropout, bandwidth limits, and latency.
(2) RGB-D Is A Rendering Plus Annotator Problem: PBR materials and HDRI domes handle appearance, while tiled rendering makes many cameras affordable, but no renderer reproduces stereo matching failure on specular and transparent surfaces unless you model it.
(3) Forces Are Solver Readouts: get_measured_joint_forces and the contact reporter expose impulses at the physics rate, so magnitudes depend on timestep, substeps, solver iterations, and contact offset rather than on any sensor.
(4) Tactile Is Reconstructed Geometry: the penetration field against the object SDF drives both the rendered gel image and the taxel forces, with shear obtained by clipping tangential displacement to the Coulomb friction cone.
(5) Physics Calibration Sets Force Fidelity: friction, mass, inertia, restitution, and SDF colliders instead of convex hulls matter more to contact realism than any amount of visual polish.
(6) Scale Comes From Trajectories, Not Pixels: Replicator randomizes lighting, materials, poses, and per-environment physics while MimicGen-style augmentation turns a few dozen human demos into hundreds of thousands of contact-rich episodes.

Figure 1: Three sensor lanes, one scene graph. Only the RGB-D lane is a genuine sensor simulation; the force lane is a solver readout and the tactile lane is reconstructed from geometry, which is why the shared realism layer does most of the sim-to-real work.

Depth is where naive pipelines fail first. The annotator gives exact metric depth, so a policy trained on it learns to trust razor-sharp object boundaries, reads valid depth off glass and polished steel, and never sees the occlusion shadow that a stereo baseline creates on the left edge of every object. A usable depth channel is therefore built by emulating the device: convert depth to disparity, quantize it, add matching noise that grows as , invalidate pixels where the simulated left and right views disagree or where the material is specular or transparent, quantize to the 1 mm uint16 grid, and finally apply the frame delay and rolling-shutter smear of the real driver. Two annotator traps show up in review: distance_to_camera returns Euclidean range while distance_to_image_plane returns the component that RGB-D APIs expect, and a policy trained on the wrong one develops a radial bias that grows toward the image corners.

Figure 2: Depth realism is arithmetic, not rendering. With px and px, a 50 mm baseline is submillimeter at 0.5 m and roughly 13 mm at 2 m, while the simulator’s raw depth sits on the flat zero-error line at every range.

Mathematical Formulation:

Where:

• is metric depth, the focal length in pixels, the stereo baseline, and the disparity in pixels.
• is subpixel matching noise, typically 0.05 to 0.2 px, and the factor is why one camera is millimeter-accurate up close and centimeter-accurate at 2 m.
• is the depth quantization step (1 mm for uint16 output) and the residual additive noise; a separate invalid mask zeroes stereo-shadow, specular, and transparent pixels.
• is the constraint impulse the solver applied at a joint or contact and the physics step, so the reported force scales as and is rate and solver dependent rather than physical.
• is the object’s signed distance function evaluated at gel sample , and is the penetration depth field on the tactile grid that drives both the gel image and the taxel normal force.
• and are normal and tangential contact force with friction coefficient ; marker flow exists only inside the cone, and slip appears when the bound is reached.

On the force side the honest framing is that PhysX gives you a well-behaved dynamics readout, not a load cell. A stiff impact resolved in one 8.3 ms step at 120 Hz appears as a single enormous impulse, whereas a real 1 kHz sensor reports a damped ringdown shaped by its own mechanical resonance plus a slowly drifting bias and a few percent of cross-axis coupling. The practical recipe is more substeps or a smaller for contact-rich phases, a contact history buffer in the sensor wrapper, then a low-pass filter to the real sensor bandwidth followed by injected bias, drift, and colored noise. Tactile needs the same discipline plus a calibration step: the elastomer in simulation is rigid geometry, so penetration depth substitutes for deformation, and the mapping from to gel pixel intensity has to be fitted against real presses on known indenters. Hysteresis, adhesion, and creep are simply absent, which bounds how far a purely simulated tactile channel can carry a slip-detection policy.

Figure 3: Tactile output is reconstructed, not sensed. The penetration field against the object SDF supplies both the rendered gel image and the taxel normal-force map, while tangential flow is only valid inside .

Property	RGB-D	Joint and contact force	Tactile
Source subsystem	RTX renderer plus Replicator annotators	PhysX 5 articulation and contact solver	SDF penetration query plus a gel renderer (TacSL, TACTO)
Native output	Exact metric depth, segmentation, normals, no holes	6D joint reaction force and net contact force at the physics rate	Penetration depth field on the gel grid, plus a shear field
Fidelity ceiling	Asset and light-transport quality; stereo and ToF failure modes are not rendered	Friction, mass, inertia, solver iterations, contact and rest offset	Gel treated as rigid geometry, so hysteresis, adhesion and creep are missing
Mandatory post-processing	Disparity quantization, noise, invalid mask, blur, frame latency	Low-pass to sensor bandwidth, bias drift, cross-axis coupling, spike clipping	Illumination and marker calibration on real presses, Coulomb-clipped shear
Real data still needed	Little for RGB, moderate for depth on shiny and transparent scenes	Small for quasi-static tasks, large for impacts and insertion	Large: a real fine-tuning set of presses, rolls and slips is usually unavoidable