DL0153 Sim-to-Real Transfer

Sim-to-real transfer is training a control policy in a physics simulator, where data is fast, safe, and infinitely resettable, and then deploying it on hardware that obeys different physics and produces different pixels. The reality gap is the mismatch between the simulated MDP and the real one, and it decomposes into at least three channels: dynamics parameters that are simply mistuned (mass, friction, joint damping, motor gains), unmodeled effects that the simulator has no term for (cable drag, gear backlash, control latency, deformable contact), and the observation gap between rendered and captured images. The three classic techniques are not competitors; each attacks a different channel. Domain randomization trains one policy over a distribution of simulators so the real robot behaves like just another sample from that distribution, system identification measures the real robot and moves the simulator’s parameters onto it, and visual domain adaptation leaves dynamics alone and instead makes sim and real images look identical to the encoder. Production pipelines for legged locomotion and dexterous manipulation compose them: identify what you can measure, randomize what you cannot, adapt the residual online with a history-conditioned policy.

(1) The Gap Is Not One Number: dynamics, unmodeled effects, and observations fail independently, so a policy can transfer perfectly in torque space and still collapse because the real camera has motion blur.
(2) Domain Randomization Widens The Simulator: sampling each episode buys robustness without any real data, provided the true parameters lie inside the support.
(3) System Identification Shrinks The Gap Itself: fitting to logged real trajectories gives a sharper simulator and a higher-performance policy, but only for effects the simulator can represent.
(4) Visual Domain Adaptation Works In Pixel Space: feature alignment, randomized-to-canonical translation, or image-to-image GANs remove appearance shift that no amount of physics tuning touches.
(5) Support Coverage Is The Binding Constraint: if falls outside the randomization support, the policy has literally never trained on the real robot’s physics and no amount of averaging helps.
(6) Online Adaptation Recovers The Lost Performance: conditioning on a short history of states and actions lets the policy infer the latent parameters at run time, which is why teacher-student distillation and rapid motor adaptation beat blind robustness.

Figure 1: The same gap, two spaces and two directions of correction. In parameter space randomization inflates the simulator’s support until it contains the real robot, while system identification translates the nominal point onto it. In feature space randomization spreads the sim distribution until it envelops the real look, while adaptation pulls real observations back onto the canonical sim appearance.

The mechanisms differ in what they need and what they cost. Domain randomization needs zero real data: you pick ranges for mass, friction, terrain roughness, motor strength, sensor noise, latency, and texture, then resample every episode so the policy cannot memorize one dynamics model. Its price is conservatism, because a memoryless policy maximizing expected return over a wide converges to behavior that is safe for the average simulator and optimal for none, which is why Automatic Domain Randomization grows the ranges only as fast as the policy can absorb them. System identification runs in the opposite direction: excite the hardware, log , and minimize prediction residuals to recover . It buys back performance and shrinks the randomization ranges you still need, but it can only fit parameters the simulator exposes, so backlash or a compliant tendon that has no corresponding term stays invisible no matter how good the fit is. Visual domain adaptation is orthogonal to both: RCAN-style translation maps heavily randomized or real images to one canonical rendering before the policy sees them, GraspGAN-style translation pushes sim images toward realism, and feature-level alignment penalizes the discrepancy between sim and real encoder activations. The standard modern recipe is a pipeline: identify → randomize the residual uncertainty → train a privileged teacher with access to → distill into a student that reads only onboard history → deploy.

Mathematical Formulation:

Where:

• and are the expected returns of policy on hardware and in simulation, and is the reality gap measured in task performance rather than in physics units.
• is the vector of simulator parameters (masses, friction coefficients, motor gains, latencies, texture and lighting seeds) and is the return in the simulator instantiated with .
• is the randomization distribution; the support condition is the precondition for transfer, since a policy trained on has no guarantee at all for parameters outside .
• is the identified parameter estimate, the simulator’s one-step transition model, and the one-step prediction residual against a logged real transition from state under action .
• indexes the real excitation trajectory; residual that cannot be driven to zero by any is unmodeled physics, and it is exactly what must be randomized or adapted away instead.
• is the visual encoder, and are observations from the two domains, and is a distribution distance such as an adversarial discriminator loss or a maximum mean discrepancy.

Figure 2: Randomization width has an inverted-U effect on real-world success while sim success decays monotonically, so sim reward is an actively misleading model-selection signal. Too narrow and sits outside the support; too wide and the policy averages over incompatible dynamics. System identification moves the useful band left by removing uncertainty you no longer need to cover.