Design the system behind Zoom AI Companion’s meeting summary: after a 45-minute video meeting, every participant receives a recap, the decisions made, and their action items within a minute or two. The pipeline must handle cross-talk, accents, and bad microphones across ten speakers, and a hallucinated action item is worse than none at all.
How would you design this system? Cover transcription, speaker attribution, the summarization stack, quality control, privacy, and cost at platform scale.
The Problem: forty-five minutes of overlapping speech from ten microphones must become a recap, a decision list, and attributed action items, within minutes, and one invented task destroys trust. Design the pipeline that was actually listening.
Answer
The design is a staged pipeline with a federated summarization stack, the approach Zoom publishes for AI Companion: streaming ASR turns audio into words, speaker diarization attributes them to people, a post-processing stage restores punctuation and domain terms, and the summarization stage drafts on a small in-house LLM, scores the draft with a learned quality model (Zoom’s Z-Scorer), and escalates to a stronger external LLM only when the score is low. The two pivotal decisions are separating “who spoke” from “what was said” (diarization plus ASR reconciled with the meeting roster, not one giant model asked to guess names) and never paying frontier-model prices for meetings a small model summarizes fine (Zoom reports comparable quality at roughly 6% of GPT-4 cost).
(1) ASR Frontend: streaming encoder-decoder (Whisper-class) with custom vocabulary biasing for product and people names; about 95% word accuracy on real meetings.
(2) Speaker Diarization: embedding-based turn detection (pyannote-style) reconciled with the participant roster and per-channel audio when available, so “Speaker 2” becomes “Ada”.
(3) Structured Summarization: chained tasks with schema-checked JSON output: chapter segmentation, recap, decisions, action items with owners.
(4) Federated Quality Control: small model drafts, a learned scorer predicts summary quality, low scores escalate to stronger LLMs, and a committee-of-LLMs cross-check suppresses hallucinated action items.
(5) Privacy by Inheritance: summaries inherit the meeting’s access control and retention; a self-hosted-only model path serves regulated customers.
Figure 1: The pipeline: hear the words, attribute them, then summarize with schema-checked structure; the scorer, not blind trust, decides when a stronger model is worth paying for.
Clarify Before Designing:
(1) Live versus Post: in-meeting “catch me up” queries (seconds, incremental) or post-meeting delivery only (minutes, batch)?
(2) Language Scope: English only, or 30-plus languages via translation assists or multilingual models?
(3) Identity Source: is a participant roster, per-channel audio, or voice enrollment available, or must speakers stay anonymous labels?
(4) Worst Error: which failure is most expensive here: hallucinated action items, missed decisions, or wrong attribution?
(5) Scale and Budget: meetings per day, average length, and the cost ceiling per meeting-minute?
Leave a Reply