DL0200 BERT GPU Underutilization

A BERT forward pass at batch 32 and sequence length 128 is a few milliseconds of dense GEMM work, so the GPU is almost never the slow part of a badly performing BERT service. Underutilization comes from three separable causes: the host-side pipeline (Python tokenization, collation, and synchronous host-to-device copies that block the training or serving loop), wasted GPU work (padding every request to max_length, batch size 1 traffic, or fragmented small kernels whose launch overhead exceeds their compute), and framework overhead (eager-mode PyTorch dispatching hundreds of tiny kernels per layer instead of fused ones). Detection is a matter of separating those causes rather than staring at the single nvidia-smi utilization number, which reports only whether any kernel was resident during a sampling window and happily shows 90% while the SMs are mostly idle. The fix ladder is ordered by cost: first hide the host work with fast tokenizers, multiple dataloader workers, pinned memory plus non_blocking copies and prefetch; then remove padding waste with length bucketing and dynamic batching; then export to ONNX for graph fusion and constant folding; then build a TensorRT engine in FP16; and only last apply INT8 post-training quantization, which is the only step in the list that can change your accuracy.

(1) Host-Bound Pipeline: a Python BertTokenizer plus per-batch collation can cost 15-25 ms while the forward pass costs 6 ms, so a serial loop pins utilization near 25% no matter how fast the GPU is.
(2) Synchronous Copies: a transfer from pageable host memory cannot be overlapped with compute, because the driver must stage it through an internal pinned buffer; pinned memory plus non_blocking=True is what makes DMA overlap legal.
(3) Padding Waste: attention cost scales as , so padding a batch whose mean length is 27 tokens out to 128 burns roughly 79% of the tokens on [PAD], which looks like high utilization and low throughput.
(4) Kernel Launch Overhead: eager BERT-base issues on the order of a thousand kernels per forward pass; at batch 1 the launch and layout overhead, not the math, sets latency.
(5) Detection By Elimination: loop over one cached pre-tokenized tensor. If throughput jumps, the bottleneck was host-side; if it does not, profile kernels with Nsight Systems and look at gaps between them.
(6) Optimize In Cost Order: dataloader and padding fixes are free and safe, ONNX and TensorRT FP16 are numerically near-lossless, and INT8 is the only step that requires a calibration set and an accuracy regression gate.

Figure 1: The same model and the same GPU, two pipelines. In the serial loop the GPU waits for tokenization and the utilization ceiling is . With three prefetching workers writing into pinned buffers, host work is fully hidden behind compute after an 18 ms warm-up and the GPU runs back to back. No kernel was made faster here; only the schedule changed.

How To Detect A Host-Side Bottleneck: start by distrusting the utilization percentage. nvidia-smi samples whether at least one kernel was resident, not how many SMs were busy, so query DCGM fields such as SM activity and SM occupancy, or run nvidia-smi dmon and watch the memory-controller column: a host-bound job shows low SM activity and near-zero copy activity in the gaps. Then run the decisive experiment, which takes ten minutes: tokenize one batch once, keep it on the device, and loop the forward pass. That measures your true kernel-only throughput, and the gap between it and end-to-end throughput is exactly your host overhead. Confirm the cause with torch.profiler (look for aten::copy_, cudaStreamSynchronize, and dataloader wait time) or an Nsight Systems timeline, where a host-bound run appears as short kernel clusters separated by long empty stretches on the CUDA stream. Common culprits found this way: the slow Python tokenizer instead of the Rust BertTokenizerFast, num_workers=0, tokenizing inside the request handler on the same thread that owns the GPU, a stray .item() or .cpu() inside the loop forcing a sync every step, and logging metrics per step.

Mathematical Formulation:

Where:

• is the fraction of wall-clock time the GPU spends executing kernels, capped at 1, with for a blocking loop and for an overlapped prefetch pipeline.
• is per-batch host time (tokenize, collate, copy) and is per-batch device time for the forward, or forward plus backward when training.
• is the number of dataloader workers or preprocessing processes; overlap only holds if their output lands in pinned memory and the copy is issued as non_blocking.
• is the dominant FLOP count per batch with the batch size, the padded sequence length, and the hidden width; the term is attention and the term is the projections and feed-forward.
• is the fraction of computed tokens that are padding, with the mean true length and the padded length; understates the true waste whenever the term dominates.

Worked Numbers For One Batch:

Three workers are enough to hide 18 ms of host work behind a 6 ms forward pass, and the same arithmetic tells you when adding workers stops helping: once drops below , extra workers only add memory pressure and page-cache churn. The padding figure is the other half of the story. Sorting a serving queue by length into buckets, or using dynamic padding to the longest member of each batch, typically recovers most of that 79% at zero accuracy cost, and it is strictly better than buying a larger GPU to compute attention over [PAD] tokens faster.

Figure 2: An illustrative optimization ladder for BERT-base at batch 32, sequence 128, on one mid-range GPU. Each rung removes a different bottleneck: host stalls first, then kernel launch and layout overhead through graph fusion, then arithmetic precision. Only the last rung can move your evaluation metric, which is why it ships behind an accuracy gate.

Code Implementation:

# 1) hide host work: fast tokenizer, workers, pinned memory, prefetch
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True)

def collate(batch):
    # dynamic padding to the longest item, not to max_length
    return tok([b["text"] for b in batch], padding=True,
               truncation=True, max_length=128, return_tensors="pt")

loader = torch.utils.data.DataLoader(
    ds, batch_size=32, collate_fn=collate,
    num_workers=4, pin_memory=True, persistent_workers=True,
    prefetch_factor=4)

for batch in loader:
    batch = {k: v.to("cuda", non_blocking=True) for k, v in batch.items()}
    with torch.autocast("cuda", dtype=torch.float16):
        out = model(**batch)      # no .item() / .cpu() inside the loop

# 2) measure kernel-only throughput to prove where the time goes
cached = {k: v.cuda() for k, v in collate(ds[:32]).items()}
torch.cuda.synchronize(); t0 = time.perf_counter()
for _ in range(200):
    with torch.no_grad(), torch.autocast("cuda", dtype=torch.float16):
        model(**cached)
torch.cuda.synchronize()
print("device-only seq/s:", 200 * 32 / (time.perf_counter() - t0))

# 3) export and build: ONNX -> TensorRT with dynamic shape profiles
# torch.onnx.export(model, tuple(cached.values()), "bert.onnx",
#                   dynamic_axes={"input_ids": {0: "B", 1: "L"}}, opset_version=17)
# trtexec --onnx=bert.onnx --fp16 --int8 --calib=calib.cache \
#         --minShapes=input_ids:1x16 --optShapes=input_ids:32x128 \
#         --maxShapes=input_ids:64x256 --saveEngine=bert.plan