DL0090 Post-Training vs Quantization-Aware Training

Post-training quantization (PTQ) takes a finished float checkpoint and converts it to low precision after the fact: a few hundred unlabeled samples are pushed through the network so the tool can observe activation ranges, pick a scale and zero point per tensor or per channel, and round the weights. No labels, no loss, no backward pass, and typically minutes to a couple of hours on a single GPU. Quantization-aware training (QAT) instead inserts fake-quantization nodes into the graph and continues training, so every forward pass sees rounded values while gradients flow through the non-differentiable rounding step via the straight-through estimator (STE). The one-line distinction worth memorizing: PTQ fits the quantizer to fixed weights, while QAT moves the weights to fit the quantizer. At INT8 on a well-behaved network the two land within a few tenths of a point of each other, so PTQ wins on cost; at 4 bits and below, or on outlier-heavy and depthwise-separable models, PTQ falls off a cliff and QAT recovers most of the loss.

(1) Where It Happens: PTQ is a post-processing step on a frozen checkpoint; QAT is a fine-tuning stage that must run inside your training pipeline with the original data loader and loss.
(2) What Data It Needs: PTQ needs only a small calibration set (roughly 128 to 1024 unlabeled samples) that is representative of deployment traffic; QAT needs labeled data, or at least a teacher model for distillation.
(3) The STE Trick: rounding has zero gradient almost everywhere, so QAT pretends the quantizer is the identity inside the clipping range and passes the gradient straight through, which is what lets weights drift toward values that round well.
(4) Where PTQ Breaks: per-tensor scales collapse when channel ranges differ by orders of magnitude (depthwise convolutions), when activations carry massive outliers (transformer residual streams), or when the bit width drops to 4 or fewer.
(5) Decision Rule: always try PTQ first because it is cheap and reversible; escalate to QAT only when a measured accuracy gap survives per-channel scales, better range selection, and bias correction.

Mechanically, a fake-quant node applies quantize → dequantize in the forward pass, so tensors stay in float during QAT but carry exactly the values the integer kernel will produce at inference. That means QAT does not speed up training; it slows it down by 20 to 40 percent while simulating the deployment numerics. The payoff is that the optimizer sees the rounding error as part of the loss surface and settles into flatter minima where a few least significant bits do not matter. PTQ has no such feedback: whatever error the rounding introduces is simply propagated forward, which is why its failure mode is a sudden collapse rather than a graceful slide.

Figure 1: The same checkpoint, two routes to integer inference. PTQ adds one forward-only calibration pass; QAT adds a full fine-tuning loop whose gradients reach the weights through the straight-through estimator.

Mathematical Formulation:

Where:

• is the dequantized value the network actually computes with, and is the original float weight or activation.
• is the scale and the zero point (the integer that maps to exactly 0.0); is the stored integer code.
• is the bit width, so gives 256 levels and only 16; are the calibrated clipping bounds, chosen per channel for weights and per tensor or per token for activations.
• is round-to-nearest and saturates values outside the representable range; both are the source of the error being managed.
• The last line is the straight-through estimator: the indicator passes the gradient unchanged inside the clipping range and zeroes it outside, which is the only reason QAT can backpropagate through rounding.
• PTQ solves for with held fixed; QAT keeps the whole chain in the graph and updates (and, with learned-step methods, itself).

Figure 2: Illustrative accuracy versus bit width. The two methods are indistinguishable at INT8, which is why PTQ dominates production INT8 pipelines; the gap opens abruptly at 4 bits and below, where the rounding error stops behaving like small additive noise.