DL0194 Sensor Degradation Feature Extractor

Degradation is not handled by making the backbone bigger. It is handled by a cheap monitoring path that runs beside the feature extractor and a policy layer that changes how the extractor’s outputs are consumed. Three monitor families are affordable per frame: referenceless image-quality and soiling heads on the raw or early-feature tensor, feature-space drift statistics such as a Mahalanobis distance between the current channel-mean vector and the training reference, and geometric self-consistency residuals such as cross-sensor reprojection error, which is the only signal that separates a genuine extrinsic shift from bad weather. Detection alone is worthless, so those signals drive three mitigations: a quality gate that reweights per-sensor features before fusion, FiLM-style conditioning that lets the shared trunk adapt its normalization to the measured degradation, and an operational-design-domain (ODD) fallback that reduces speed, disables a fused output, or triggers a wiper and an online extrinsic re-estimation. The pipeline therefore reads capture → ISP → undistort → backbone → gate → fusion → heads, with the monitors tapping the first three stages and writing only into the gate.

(1) Separate Detection From Correction: a small monitor head that answers “how bad is this input” is far easier to train and validate than a backbone expected to be silently invariant to everything. It also gives you an auditable signal to log.
(2) Three Complementary Signals: pixel-level quality catches soiling and droplets, feature drift catches global appearance shift like rain veiling or blooming, and reprojection residuals catch calibration error. No single one covers all three failure classes.
(3) Geometry For Geometry Faults: a 0.5 degree extrinsic yaw drift leaves every image looking perfectly clean, so appearance-based monitors are blind to it; only cross-sensor residuals or photometric alignment expose it.
(4) Gate The Fusion, Not The Backbone: down-weighting a corrupted branch with normalized weights is a one-line change at inference and needs no retraining, whereas swapping backbone weights per weather condition doubles the validation surface.
(5) Train For Dropout, Not Just For Rain: corruption augmentation plus random modality dropout is what makes a gated fusion model usable when a branch is masked, otherwise the fused head has never seen a zeroed input.
(6) Absolute Scores Plus An Abstention Path: a softmax gate can only express relative trust, so a separate calibrated absolute quality score must be able to declare that all sensors are bad and hand control to the ODD layer.

Figure 1: The monitors sit outside the critical path and write only into the gate. An alarm changes the fusion weight , the FiLM conditioning, and the declared ODD, but it never edits backbone weights at runtime, which keeps the deployed model bit-identical and the failure behaviour testable.

Each monitor has a different latency and a different false-alarm profile. The soiling and quality head is a per-tile classifier over an early feature map, typically under 1 ms on an embedded accelerator, and it is the only monitor fast enough to drive a physical actuator such as a nozzle or a heater. Feature drift is computed from the channel means of a mid-level tensor against a reference mean and covariance collected on clean data, then smoothed by a CUSUM accumulator so that a single dark frame does not raise an alarm while a sustained shift does within a few frames. Reprojection residuals need matched features across overlapping fields of view or LiDAR points projected into the image, so they run at a lower rate over a sliding window of several seconds, which is acceptable because extrinsic drift from thermal expansion or a curb strike is either slow or a step change that persists. Lens distortion sits between these cases: an intrinsics change makes straight lines curve and inflates residuals even for a single camera, and the correct response is to re-estimate the undistortion look-up table rather than to touch the network, because a CNN trained on rectified images treats a mis-rectified frame as out-of-distribution geometry.

Figure 2: The two monitors have orthogonal signatures. Rain moves the feature-drift statistic and leaves the reprojection residual untouched, while a mount shift moves the residual by 2.5 px without disturbing appearance statistics at all. Reading only one monitor guarantees you misdiagnose one of the two faults.

Quality-Gated Fusion:

Where:

• is the fused feature tensor consumed by the task heads, and is the raw feature map of sensor , with its degradation-conditioned version.
• is the absolute quality score produced by a small monitor head with logistic output ; it is supervised by synthetic corruption labels and calibrated on held-out real degraded clips.
• and are the FiLM scale and shift vectors, and is channel-wise multiplication.
• is a content-dependent attention logit from the ordinary fusion module, so the gate combines what is informative with what is trustworthy.
• sums to 1 over sensors , which is why a low but uniform across all sensors must be caught by the raw values rather than by the weights.

Runtime Degradation Monitors:

Where:

• is the Mahalanobis drift at frame between the current channel-mean vector of a mid-level feature map and the clean-data reference with covariance .
• is the CUSUM statistic with slack , which is set just above the clean-condition mean of ; an alarm is raised when and is then reset to 0.
• is the reprojection residual in pixels for correspondence , where is the observed image point, the 3D point from LiDAR or a second camera, the extrinsic transform, and the projection using current intrinsics.
• A robust percentile of above roughly 1 px sustained over a window indicates extrinsic or intrinsic drift rather than matching noise, and triggers online recalibration.

The handling policy has to be trained for, not bolted on. A gate that can zero a camera branch is only safe if the fused head saw randomly dropped modalities and heavy corruption augmentation during training, otherwise masking a branch pushes the fusion layer into a region it never visited and accuracy collapses harder than with the corrupted input left in place. Augmentation alone is also insufficient, because it buys average-case robustness while a gate buys graceful worst-case behaviour: at high rain severity the model that can lean on LiDAR keeps far more of its mAP than the model that must average a clean point cloud with a veiled image. The cost is roughly one mAP point in clean weather, from the gate occasionally distrusting a good camera, plus the engineering burden of calibrating so that the gate does not permanently learn to ignore a sensor after a single bad deployment week.

Figure 3: Corruption augmentation flattens the curve, but only the quality gate changes the shape of the tail, because it can stop trusting the camera entirely. The 1-point clean-weather cost at severity 0 is the price of that option, and it is the number a reviewer should ask you for.