DL0187 Hardware-Aware Neural Architecture Search

NAS automates architecture design by specifying three things: a search space of candidate operations and connectivity, a search strategy that proposes architectures, and an evaluation method that scores them. Classical NAS maximizes validation accuracy alone, which reliably produces models that are unusable on a phone because accuracy is monotone in capacity. Hardware-aware NAS changes the objective rather than the search algorithm: the target device becomes part of the reward, either as a hard constraint on measured latency or as a multi-objective term that trades accuracy against cost. The critical detail is that FLOPs and parameter counts are poor latency proxies, because depthwise convolutions, squeeze-excite blocks, and grouped convolutions are memory-bandwidth bound rather than compute bound, so the cost signal must come from on-device measurement or a layer-wise latency lookup table (LUT) calibrated on the actual CPU, GPU, DSP, or NPU. Memory enters as a separate ceiling on peak activation footprint (the binding constraint on microcontrollers with a few hundred KB of SRAM), and energy enters through measured joules per inference, which tracks DRAM traffic more than arithmetic. Because a full search costs thousands of GPU-hours per device, production systems amortize it with a weight-sharing supernet that is trained once and then queried per deployment target.

(1) Three Components: search space, search strategy (reinforcement learning, evolution, or gradient-based), and evaluation. The space dominates the outcome, since a badly chosen space caps the achievable Pareto front no matter how good the optimizer is.
(2) FLOPs Are Not Latency: two blocks within 5% of each other in FLOPs can differ by 1.8x in measured milliseconds, so proxy metrics silently select the wrong architecture.
(3) Cost In The Objective: either a hard constraint , a soft reward such as MnasNet’s , or a differentiable expected latency term added to the loss.
(4) Latency Lookup Tables: profile each candidate operator once per device and sum per-layer times, which makes the cost model cheap and differentiable but blind to operator fusion and thermal throttling.
(5) Memory And Energy Are Separate Constraints: peak activation RAM bounds what fits in SRAM, and energy per inference is dominated by off-chip memory traffic, so neither is implied by a latency target.
(6) Amortization Is The Production Trick: a once-for-all supernet trained with progressive shrinking yields deployable subnets for a new phone in minutes instead of a fresh search per device.

Figure 1: The loop is sample subnet → score accuracy on a weight-sharing supernet → query a hardware cost model → combine into one multi-objective reward → update the sampler. Only the cost model is device-specific, which is why the same search pipeline produces a different winner for a mid-range CPU than for an NPU.

The cost model is where most engineering effort goes. A layer-wise LUT stores the measured runtime of every candidate operator at every feature-map shape on the target, so predicting a candidate’s latency is a table sum instead of a deployment. This is fast enough to put inside a training loop, and because the sum is linear in the per-operator times it can be made differentiable with respect to architecture probabilities. The failure mode is LUT drift: a compiler that fuses convolution with batch normalization and activation, a scheduler that changes clock frequency under load, or a runtime that picks a different kernel for a specific channel count will all break additivity, and errors compound across dozens of small layers. Practical pipelines therefore validate the LUT against end-to-end on-device measurements for a sample of candidates and refit when the mean absolute percentage error drifts past a few percent.

Figure 2: Measured latency versus FLOPs for candidate blocks on one mobile CPU. The ordering is not preserved: a grouped convolution and a squeeze-excite MBConv cost roughly 1.9x and 1.8x the latency of blocks with nearly identical FLOPs, because both are memory-bandwidth and kernel-launch bound. Searching against FLOPs would pick the slow block.

Mathematical Formulation:

Where:

• is a candidate architecture drawn from the search space , and are its weights, either trained from scratch or inherited from a supernet.
• is the latency on target hardware and the deployment budget, so the same has a different cost on a CPU, a DSP, and an NPU.
• is the soft multi-objective reward, with exponent negative (MnasNet uses ) so that exceeding the budget is penalized smoothly rather than rejected outright.
• is the differentiable alternative: cross-entropy on weights plus a log-latency regularizer weighted by , which controls where on the Pareto front the search lands.
• is the softmax probability of choosing operator at layer , and the measured LUT entry for that operator at that layer’s shape, so expected latency is linear in and its gradient is exactly .
• is the activation tensor size at layer ; since an inplace-scheduled runtime holds an input and an output buffer simultaneously, the peak pair sum must fit the on-chip budget .

Figure 3: The same architecture set has a different Pareto front per device. Architecture Y misses a 20 ms budget on the CPU at 26 ms but clears it on the NPU at 16 ms, so the optimal pick flips. This is why hardware-aware NAS is rerun (or a supernet requeried) for each target rather than solved once.

Budget Decomposition For A 30 ms Camera Frame:

The search target is never the product-level frame budget. Resize, color conversion, and normalization consume a few milliseconds before the network runs, and non-maximum suppression or decoding consumes more afterwards, so the network budget that goes into the reward is what remains. Getting this wrong by 5 ms produces an architecture that is Pareto-optimal against the wrong constraint, which is a more common production failure than a weak search algorithm.