What is quantization (FP16/BF16/INT8/INT4), and how does it affect model memory and accuracy?
Answer
Quantization stores weights, activations, or KV-cache entries in a numeric format with fewer bits than the FP32 values the model was defined in, trading numerical resolution for memory and bandwidth. Memory falls almost exactly linearly in bit width: a 70B-parameter model occupies 280 GB in FP32, 140 GB in FP16 or BF16, 70 GB in INT8, and roughly 37 GB in 4-bit with group scales. The two 16-bit floating formats differ only in how they split the 16 bits: FP16 spends 5 bits on the exponent and 10 on the mantissa (max magnitude 65,504, so gradients can overflow without loss scaling), while BF16 keeps FP32’s 8 exponent bits and truncates the mantissa to 7, giving the full 
dynamic range at coarser resolution. Below 16 bits the formats become integers plus a scale, so a real number is recovered as 
, and quality now depends on how finely those scales are estimated. In practice INT8 weights are essentially free in accuracy with per-channel scales, INT4 weight-only costs a few tenths of a perplexity point on a 70B model with a good calibration method, and quantizing activations is the hard part because transformer activations contain a handful of channels with magnitudes 20x to 100x the median.
(1) Exponent Versus Mantissa: FP16 and BF16 use the same 16 bits, but BF16’s 8 exponent bits buy dynamic range at the cost of precision, which is why BF16 is the default training format on modern accelerators and needs no loss scaling.
(2) Memory Is Linear In Bits: parameter memory is 
bytes, so each halving of the bit width halves the checkpoint, the resident weights, and the bytes moved per token.
(3) Integers Need A Scale And A Granularity: per-tensor scaling is cheapest, per-channel is the practical minimum for weights, and group-wise (typically 128 weights per scale) is what makes 4 bits usable.
(4) Weight-Only Helps Because Decode Is Bandwidth-Bound: single-stream generation reads every weight per token, so 4-bit weights cut latency even when the matmul itself runs in FP16 after dequantization.
(5) Error Concentrates In Outliers: a single extreme value stretches the scale and destroys resolution for every other value in the group, which is why outlier handling matters more than the nominal bit count.
Figure 1: Bit budgets drawn to scale. BF16 keeps FP32’s 8 exponent bits and pays with 7 mantissa bits, FP16 does the reverse, and the sub-8-bit formats drop the exponent entirely in favour of an external scale factor.
An integer format is defined by an affine map from the real line onto 
evenly spaced levels. Symmetric quantization fixes the zero-point at 
and is the standard choice for weights, whose distribution is roughly zero-centred; asymmetric quantization keeps a learned 
and suits post-ReLU or post-GELU activations that sit mostly on one side of zero. The granularity of 
is the real design knob: one scale per tensor stores nothing extra but is destroyed by a single outlier channel, whereas one scale per group of 
weights adds a 16-bit scale and a 16-bit zero-point per group, which raises an INT4 tensor from 4.0 to 4.25 effective bits. Calibration methods such as GPTQ and AWQ then choose the rounding of each weight to minimise the error of the layer output on a small calibration set rather than the error of the weight itself, which is what closes most of the remaining gap at 4 bits.
Mathematical Formulation:
Where:
is the dequantized value actually used in the matmul, and 
is the original FP32 or BF16 weight.
is the stored integer code, clipped into 
; values outside the calibrated range are saturated rather than wrapped.
is the scale (step size) and 
the zero-point; symmetric quantization sets 
and uses 
.
is the nominal bit width and 
the group size, the number of weights sharing one 
pair stored in FP16, hence the 
overhead.
is the effective bits per weight; with 
and 
this gives 
.
is the parameter count and 
the weight memory in bytes; for 
this is about 37 GB. Required condition: must be computed from calibration data before any activation quantization is applied.
is the dequantized value actually used in the matmul, and 
is the original FP32 or BF16 weight.
is the stored integer code, clipped into 
; values outside the calibrated range are saturated rather than wrapped.
.
is the nominal bit width and 
the group size, the number of weights sharing one 
pair stored in FP16, hence the 
overhead.
is the effective bits per weight; with 
and 
.
is the parameter count and 
the weight memory in bytes; for 
this is about 37 GB. Required condition: 
must be computed from calibration data before any activation quantization is applied.
Figure 2: Weight memory for a 70B model. Only the 4-bit variant fits one 80 GB GPU with room left for the KV cache, and because decoding is memory-bandwidth bound the same reduction shows up directly in tokens per second.
| Property | FP16 | BF16 | INT8 | INT4 (g=128) |
|---|---|---|---|---|
| Layout | 1 sign, 5 exponent, 10 mantissa | 1 sign, 8 exponent, 7 mantissa | 8-bit integer plus a per-channel FP16 scale | 4-bit integer plus a scale per 128 weights |
| Dynamic range | Max 65,504; underflows below 6e-5 | Same as FP32, about 3.4e38 | 256 levels inside the calibrated range | 16 levels per group; clipping is the main risk |
| 70B weights | 140 GB | 140 GB | 70 GB | About 37 GB at 4.25 effective bits |
| Typical role | Inference, and training with loss scaling | Default training and master-weight format | W8A8 serving at high batch, compute-bound prefill | Weight-only decode on capacity-limited GPUs |
| Accuracy cost | None measurable for inference | None measurable; the lost mantissa bits rarely matter | Under 1% with per-channel scales and outlier handling | A few tenths of perplexity on 70B, clearly worse below 7B |
| Hardware support | Tensor cores on every recent GPU | Ampere onward, plus TPUs | Roughly 2x the FP16 matmul throughput | No native 4-bit matmul on most GPUs; kernels dequantize in registers |
Leave a Reply