DL0156 Force-Torque and Visual Fusion in VLA

Fusion begins with tokenization, not with attention: a 6-axis wrench sampled at 1 kHz and the per-joint motor currents are cut into short windows (typically 50-200 ms), pushed through a small 1D CNN or spectrogram encoder, pooled down to roughly one token per 10 ms, and projected by a learned matrix into the same -dimensional space that holds the visual patch tokens. Inside the block, the language and vision residual stream supplies the queries while the concatenated visual and proprioceptive tokens form the key/value memory, so softmax attention decides layer by layer how much contact evidence to read instead of averaging modalities with fixed weights. Because the pretrained VLM has never seen a wrench token, the cross-attention output is normally added through a zero-initialized tanh gate, which makes the new modality an exact no-op at initialization and lets it grow in during robot fine-tuning without destroying the language prior. The two streams also carry different information: vision says where and what, force says when and how hard, and force is the only channel that survives occlusion once the tool is inside the hole. What this fusion cannot do is close the contact loop, since one VLA forward pass costs 50-200 ms while an impact transient decays in a few milliseconds, so the policy emits setpoints, target wrenches, and impedance gains for a 1 kHz low-level controller that does the actual reacting.

(1) Window Then Tokenize: a 1 kHz stream cannot enter a 5 Hz backbone sample by sample, so it is windowed and pooled into about 10 tokens per 100 ms, which keeps the added sequence length near 2 percent of a two-camera observation.
(2) Shared Embedding Space: a per-modality encoder plus a linear projection is what makes wrench tokens, current tokens, and patch tokens legal keys in the same attention operation.
(3) Queries From The Stream, Keys From The Sensors: cross-attention is asymmetric on purpose, so the policy can ignore force during free-space motion and attend heavily to it during contact.
(4) Zero-Init Gating: with preserves the pretrained VLM at step zero and is the standard fix for adding a modality that was absent from web-scale pretraining.
(5) Action Space Must Be Contact-Aware: position-only outputs cannot express compliance; the head has to emit a target wrench or stiffness and damping for an impedance or admittance law.
(6) Frequency Separation: the transformer supervises at 5-10 Hz and the 1 kHz controller reflexes, because policy latency is roughly 30 times longer than the impact transient it would need to catch.

Figure 1: Only the encoders and projections are modality-specific. After projection, the wrench and current tokens are ordinary keys and values, and the zero-initialized gate controls how much of that memory reaches the residual stream. The head deliberately outputs gains and wrenches, not just poses, because a stiff position command in contact is how robots break parts.

Two practical details decide whether this works on hardware. The first is signal conditioning: a raw wrench includes tool gravity, payload inertia, and thermal bias drift, so the encoder must be fed a gravity-compensated and re-zeroed signal, otherwise the same physical contact produces different tokens after every payload change. Joint current is an even noisier torque proxy, since holds only after subtracting harmonic-drive friction and stiction, which are velocity-dependent and hysteretic; current is therefore best used for coarse collision and jam detection across the whole arm rather than for fine wrench estimation at the tool. The second is bandwidth. Slip, chatter, and impact live in the 100-1000 Hz band, so pooling to 10 Hz destroys exactly the evidence you added the sensor for, which is why many implementations feed short-time spectral features or per-window statistics (peak, RMS, jerk) alongside the mean rather than a naive average.

Figure 2: The rate mismatch is the whole design constraint. The impact transient starts and finishes inside a single policy cycle, and the camera may not even sample it, so the VLA can only choose a compliance policy in advance and let the 1 kHz loop execute it. Anything that must react in under 10 ms cannot live in the transformer.

Mathematical Formulation:

Where:

• is the wrench at time , with three forces and three moments expressed in the tool frame after gravity and payload compensation.
• is the measured current of joint , the motor torque constant, and the friction and stiction term that makes current a biased torque estimate.
• is the window encoder over the last samples and its projection to the backbone width ; is the resulting small set of soft tokens.
• is the concatenated key/value memory holding visual, force-torque, and joint-current tokens, and is the language and vision hidden state that provides queries through .
• is the learned scalar gate, initialized at so that and the block reduces to the pretrained VLM on the first step.
• and are the commanded stiffness and damping matrices, and the desired pose and velocity, and the wrench the 1 kHz controller actually applies.

Rate And Token Budget:

A 100 ms window at 1 kHz gives 100 raw samples, pooled to 10 tokens, so about 100 force tokens per second ride alongside thousands of visual tokens. Meanwhile roughly 29 impact transients fit inside one policy forward pass, which is the quantitative reason the fusion is about situational awareness and gain scheduling rather than reflexes. The most common training pathology follows from the same asymmetry: proprioceptive inputs are low-dimensional and almost perfectly predict the next demonstrated action, so behaviour cloning happily learns to regress the previous command from state and stops looking at the camera. That is textbook causal confusion, and the usual defenses are proprioception dropout, delta actions instead of absolute targets, action chunking, and holdout episodes with the object moved.

Figure 3: Illustrative attention-mass trace for a peg-insertion policy. During free-space reaching the wrench tokens are near zero and attract almost no mass, and once the peg is inside the hole vision is largely occluded so the wrench becomes the dominant evidence. The shares do not sum to one because language and history tokens hold the remainder.