Design a small LLM that runs entirely on a phone for a privacy-preserving assistant, in the style of Apple Intelligence’s on-device foundation model or Gemini Nano on Pixel. The model must fit in roughly 2-4 GB of RAM, respond in under 300 ms on a mobile NPU, and work with the radio off.
It also has to stay polite. It should refuse harmful requests, avoid toxic or biased outputs, and hold a respectful tone even when the user is deliberately provoking it. Battery life and offline capability matter as much as raw benchmark quality, and no user text may leave the handset.
How would you design this system? Cover model compression and quantization for the NPU, the alignment strategy that keeps the model polite at a small scale, the on-device serving path, and how you evaluate safety without sending user data to the cloud.
The Problem: a few gigabytes of RAM, a fraction of a second, no network, and a user who is testing how rude the assistant will get.
Answer
The design is a distilled 3B-parameter model compressed with quantization-aware training to about 2 bits per weight, served on the NPU with per-feature LoRA adapters and a tiny always-resident guard model. Politeness is trained in rather than prompted in: a large aligned teacher supplies refusal and tone supervision, a preference pass sharpens “polite but not preachy”, and safety data stays inside the quantization loss so alignment survives compression. Two decisions are pivotal. First, safety is re-verified after quantization, because low-bit weights erode refusal behavior faster than they erode average benchmark quality. Second, a separate 100M-parameter guard screens the prompt and the streamed output, so a single jailbroken generation is caught by a component that can be patched over the air without retraining the base model.
(1) Distill + QAT + Adapters: prune and distill a large teacher into ~3B parameters, then quantization-aware train to 2-4 bits and ship 16-bit LoRA adapters per feature (summarize, reply, rewrite).
(2) Scratch-Trained Small Model + PTQ: train a 1-2B model on curated data and apply post-training 4-bit quantization; simplest pipeline, weakest quality per byte.
(3) Elastic Nested Model: a MatFormer-style checkpoint whose submodels can be extracted at different sizes, letting the runtime trade quality for RAM under memory pressure.
(4) On-Device Draft + Private Cloud Escalation: answer locally by default and escalate hard prompts to a privacy-hardened server tier with explicit consent.
Figure 1: Everything inside the dashed boundary runs on the handset: guard, decode loop, adapters, and cache. Nothing crosses the boundary, so every check has to happen before pixels appear.
Clarify Before Designing:
(1) Latency Definition: is 300 ms the time to first token or the full reply? Streaming a 60-token answer at 30 tokens per second takes two seconds no matter how good the prefill is.
(2) Task Set and Context: which features ship (reply suggestions, summarization, rewriting), and what is the maximum prompt length? Prefill cost and KV cache size follow directly from that number.
(3) Device Floor: which chip generation is the minimum, and what does its NPU actually execute natively (INT8, INT4, palettized weights, per-channel scales)?
(4) Escalation Policy: is any server path permitted with consent, or is strict offline operation a product promise we cannot break?
(5) Safety Bar: which refusal taxonomy applies, and what false-refusal rate on benign-but-edgy prompts is acceptable? Over-refusal is the failure users complain about.
(6) Telemetry Rules: may we collect differentially private aggregate counters and opt-in donated conversations, or literally nothing?
Leave a Reply