Category: Easy

DL0096 Autoencoders Practical Uses

What is an autoencoder? What are the practical uses of autoencoders?

Answer

An autoencoder is a network trained to reconstruct its own input through a narrow bottleneck: an encoder maps $x \in \mathbb{R}^{D}$ to a code $z \in \mathbb{R}^{d}$ with $d \ll D$ , and a decoder maps that code back to $\hat{x}$ , with the reconstruction error as the only supervision. Because the target is the input itself, training needs no labels, which is why autoencoders are the canonical self-supervised compression model. The bottleneck is what makes the code useful: the network cannot pass everything through, so it must keep the structure that is shared across typical inputs and drop the rest. With linear encoder and decoder and squared loss, the optimum spans exactly the same subspace as the top $d$ principal components; nonlinear layers buy a curved manifold instead of a flat one. The practical uses split into three families depending on which part of the trained model you keep: the error (anomaly detection), the code (compression, retrieval, visualization, latent space for a diffusion model), or the decoder (denoising, inpainting, generation once the latent is regularized).

(1) Bottleneck Plus Reconstruction Loss: the objective is $\|x - \hat{x}\|_2^2$ (or cross-entropy for binary pixels), and the capacity constraint, not the loss, is what forces a compressed representation.
(2) Undercomplete Versus Regularized: an undercomplete model constrains $d$ directly, while sparse, denoising, and contractive variants keep a wide code and constrain it with an $L_1$ penalty, input corruption, or a Jacobian penalty.
(3) Linear Case Recovers PCA: a single linear layer each side with MSE learns the principal subspace, so any gain over PCA must come from nonlinearity, not from the architecture being a neural network.
(4) Anomaly Detection By Error Score: train only on normal data, then score new samples with $s(x) = \|x - \hat{x}\|_2^2$ and flag $s(x) > \tau$ , with $\tau$ set from a percentile of validation errors.
(5) Compression Into A Working Latent Space: modern latent diffusion models run denoising inside an autoencoder latent that shrinks a $512 \times 512 \times 3$ image to $64 \times 64 \times 4$ , a 48x reduction in elements, which is what makes high-resolution sampling affordable.
(6) Not Generative By Default: a vanilla autoencoder learns no density over $z$ , so decoding an arbitrary code usually yields garbage; sampling requires a VAE, VQ-VAE, or a prior fitted over the codes afterwards.

Diagram of an autoencoder: a 784-dimensional input passes through a 256-unit hidden layer into a 32-dimensional code, then back out through a 256-unit hidden layer to a 784-dimensional reconstruction, with both ends feeding a reconstruction loss box

Figure 1: The flow is $x$ → $z$ → $\hat{x}$ ; the $784 \rightarrow 32$ bottleneck is a 24.5x squeeze, and nothing but the reconstruction error supervises what the code contains.

Where do they actually earn their place in a system? Anomaly detection on sensor, telemetry, and transaction data is the most common production use, because normal data is abundant and labeled failures are not. Denoising and restoration use the decoder: train with corrupted inputs and clean targets, and the model learns to project back onto the data manifold. Dimensionality reduction gives a cheap nonlinear preprocessor, typically compressing to 32 or 64 dimensions before a nearest-neighbor index or a t-SNE plot, which is far cheaper than running the visualization on raw features. Two uses dominate current deep learning: the perceptual autoencoder that provides the latent space for latent diffusion, and masked autoencoders, where a ViT reconstructs 75% masked patches as a pretraining task and the encoder is then fine-tuned for classification or detection. Recommender systems also use the pattern directly, reconstructing a user’s sparse interaction vector to predict the entries that are missing.

Histogram of reconstruction errors: normal samples cluster at low error, anomalous samples spread into a long right tail, with a vertical threshold line drawn at the 99th percentile of the normal validation errors

Figure 2: Illustrative anomaly scores from an autoencoder trained on normal data only. A threshold at the 99th percentile of normal validation errors fixes the false-positive rate at 1% by construction, and the achievable recall depends entirely on how far the anomalous tail separates.

Mathematical Formulation:
$z = f_{\theta}(x)$
$\hat{x} = g_{\phi}(z)$
$\mathcal{L}(\theta, \phi) = \frac{1}{N} \sum_{i=1}^{N} \| x_i - \hat{x}_i \|_2^2$
$s(x) = \| x - g_{\phi}(f_{\theta}(x)) \|_2^2$
$\mathcal{L}_{\mathrm{DAE}} = \mathbb{E} \| x - g_{\phi}(f_{\theta}(\tilde{x})) \|_2^2$

Where:

$z \in \mathbb{R}^{d}$ is the code and $\hat{x} \in \mathbb{R}^{D}$ the reconstruction of the input $x \in \mathbb{R}^{D}$ .
$D$ is the input dimension and $d$ the code dimension, with $d \ll D$ in the undercomplete case.
$f_{\theta}$ is the encoder and $g_{\phi}$ the decoder, with trainable parameters $\theta$ and $\phi$ optimized jointly.
$i \in \{1, \ldots, N\}$ indexes the $N$ training examples, and $\mathcal{L}$ is the mean squared reconstruction loss.
$s(x)$ is the anomaly score, and a sample is flagged when $s(x) > \tau$ for a threshold $\tau$ chosen from a high percentile of normal validation scores.
$\tilde{x}$ is a corrupted copy of $x$ (additive noise, masking, or dropout on inputs); the denoising objective still targets the clean $x$ , which is what prevents the identity solution even when $d \geq D$ .

Property	Undercomplete autoencoder	PCA	VAE
Mapping	Nonlinear encoder and decoder, learned by SGD	Single linear projection, closed-form via SVD	Nonlinear, but the encoder outputs a distribution over z
Objective	Reconstruction error only	Maximum retained variance, equivalent to linear MSE	Reconstruction plus a KL term pulling z toward a prior
Latent geometry	Arbitrary, often with holes and unused directions	Orthogonal ordered axes, fully interpretable scale	Smooth and roughly isotropic, so interpolation is meaningful
Sampling new data	Not supported without fitting a prior over codes afterwards	Only under an explicit Gaussian model such as probabilistic PCA	Yes, sample the prior and decode
Typical use	Anomaly scoring, denoising, compact features for indexing	Fast baseline, whitening, exploratory analysis	Latent space for diffusion, generative sampling, recsys
Main caveat	Too much capacity collapses to near-identity and the code stops being informative	Cannot represent curved manifolds; cost grows with feature count	Blurry reconstructions and posterior collapse if the KL weight is too high

August 16, 2026

DL0095 Zero-Shot vs Few-Shot

How do zero-shot and few-shot prompting differ, and when does few-shot prompting beat fine-tuning?

Answer

Both are inference-time conditioning: the weights $\theta$ never change, only the tokens placed before the query. Zero-shot prompting gives an instruction and the input, so the model must map the task description onto a behavior it already learned during pretraining or instruction tuning. Few-shot prompting (in-context learning) prepends $K$ solved demonstrations $(x_i, y_i)$ that pin down the output format, the label space, and the input distribution before the real query arrives, which is why it helps most on tasks with an unusual schema or a strict output contract. In the original GPT-3 study, TriviaQA accuracy for the 175B model moved 64.3% → 71.2% going from zero-shot to 64-shot, and the gap between the two settings shrinks as models get better instruction tuning. Few-shot prompting beats fine-tuning when labeled data is scarce (roughly tens of examples), the task spec is still changing weekly, one frozen base must serve many tasks, or no training infrastructure exists; fine-tuning wins once you have thousands of clean labels, need the lowest possible per-request cost and latency, or need behavior that no prompt reliably elicits. The decision is mostly economics plus label count, not model quality: demonstrations are paid for on every request, while a fine-tune is a one-time cost amortized over traffic.

(1) Same Weights, Different Context: neither method computes a gradient; few-shot differs from zero-shot only by the demonstration block $S_K$ inserted into the prompt.
(2) Demonstrations Teach Format, Not Mostly Facts: the label space, input distribution, and output template drive most of the gain, which is why even partly incorrect labels in the exemplars often still work.
(3) Cost Is Recurring: $K$ exemplars add $K T_{ex}$ prefill tokens per call, inflating time-to-first-token and input spend on every request forever.
(4) Fine-Tuning Trades Setup For Marginal Cost: a LoRA run costs money once and then serves a short prompt, so it wins above a traffic break-even point.
(5) Data Volume Decides The Ceiling: with a handful of labels in-context learning is usually ahead; with thousands, parameter updates reach accuracy no prompt matches.

The mechanism is worth stating precisely because it predicts the failure modes. Demonstrations act as a task locator rather than a training set: replacing gold labels with random ones from the correct label set degrades few-shot accuracy far less than removing the labels entirely, which shows the exemplars are mostly specifying which distribution to condition on. That same conditioning creates strong biases: majority-label bias (a class over-represented in the exemplars gets over-predicted), recency bias (the last exemplar dominates), and ordering sensitivity that can swing accuracy by tens of points across permutations of the same $K$ examples. Calibration on a content-free input and stratified, order-shuffled exemplar selection recover most of that variance, and any prompt tuned on a large validation set is no longer honestly “few-shot” because the selection itself consumed labels.

Line chart of task accuracy against the number of in-context examples for an 8B and a 70B frozen model, with two dashed horizontal reference lines for an 8B model fine-tuned on 100 and on 5000 labels

Figure 1: Illustrative shot-scaling behavior: most of the in-context gain arrives by $K = 4$ and flattens after $K = 16$ , a model fine-tuned on only 100 labels sits near the few-shot curve, and 5,000 labels put the fine-tuned small model above the frozen large model.

Mathematical Formulation:
$p_{\theta}(y \mid I, x)$
$p_{\theta}(y \mid I, S_K, x)$
$S_K = ((x_1, y_1), \ldots, (x_K, y_K))$
$T_{ctx} = T_I + K T_{ex} + T_x$
$40 + 32 \times 60 + 30 = 1990$
$R^{*} = \frac{C_{ft}}{K T_{ex} c_{in}}$

Where:

$y$ is the generated answer, $x$ the query, and $I$ the instruction text; the first two lines are the zero-shot and few-shot predictive distributions under identical $\theta$ .
$S_K$ is the demonstration block and $K$ the shot count, with $K = 0$ recovering the zero-shot case exactly.
$T_I$ , $T_{ex}$ , and $T_x$ are token lengths of the instruction, one exemplar, and the query; $T_{ctx}$ is the prefill length that sets time-to-first-token.
The numeric line instantiates $T_{ctx}$ for $K = 32$ exemplars of 60 tokens each, giving 1,990 prompt tokens against 70 for the zero-shot prompt.
$c_{in}$ is the price per input token, $C_{ft}$ the one-time fine-tuning cost, and $R^{*}$ the break-even request volume above which the fine-tune is cheaper; it assumes both options serve the same output length and per-token price.

Plugging in real numbers makes the trade-off concrete. At $c_{in} = \$0.30$ per million input tokens, the 1,920 extra tokens from 32 exemplars cost about $\$0.00058$ per request, so a $\$60$ LoRA job pays for itself after roughly 104,000 requests. Prompt caching changes that arithmetic sharply: because the exemplar block is a fixed prefix, cached reads billed near 10% of the input rate push the break-even beyond a million requests and also cut the prefill latency penalty. That is why the honest answer to “few-shot or fine-tune” depends on traffic volume, prefix stability, and whether your serving stack caches, not on which technique sounds more advanced.

Line chart of cumulative extra cost in dollars against requests served, comparing few-shot prompting with and without prompt caching against a flat one-time fine-tuning cost, with the crossover marked near 104 thousand requests

Figure 2: Illustrative cost crossover: the few-shot line grows linearly with traffic because the exemplar prefix is re-billed per call, the fine-tune is a flat one-time charge, and prompt caching flattens the few-shot slope by roughly an order of magnitude.

Property	Zero-shot	Few-shot (in-context)	Fine-tuning (LoRA)
Labeled examples needed	None, only a clear instruction	Typically 4 to 64, plus a small set for prompt selection	Hundreds to tens of thousands
Where task knowledge lives	Pretraining and instruction tuning only	In the prompt, re-sent or cached per request	In adapter weights, prompt stays short
Prompt tokens per request	70 in the worked example	1,990 at K=32, so higher time-to-first-token	70, same as zero-shot
Iteration speed	Seconds, edit the instruction	Seconds, swap or reorder exemplars	Hours per run plus eval and deploy
Main failure mode	Wrong output schema, task misread	Majority-label and recency bias, ordering variance, context limits	Overfits small or noisy label sets, forgets off-task behavior
Multi-task serving	One model, one endpoint	One model, per-task prompt template	One adapter per task over a frozen base

August 16, 2026

DL0085 RAG vs Fine-Tuning

When should you use RAG instead of fine-tuning?

Answer

Reach for RAG when the failure is a knowledge gap, and fine-tune when the failure is a behavior gap. Knowledge gaps look like this: the required facts change (pricing, inventory, incident history), they are private to a tenant, they are too numerous to memorize (millions of documents), or the answer must carry a citation a human can audit. Behavior gaps look different: the model has the right context in the prompt and still emits the wrong JSON schema, the wrong tone, the wrong clinical or legal register, or a malformed tool call. The diagnostic that settles most arguments is a context-injection test: paste the gold document into the prompt by hand, and if the answer becomes correct, you have a retrieval problem, not a weights problem. Update economics reinforce the split, because a corrected fact ships to a RAG system in minutes by reindexing one chunk, while the same correction in a fine-tuned checkpoint means a training run plus a full regression eval, and controlled comparisons have repeatedly found that continued pretraining on new documents injects facts less reliably than simply retrieving them.

(1) Knowledge vs Behavior Test: if hand-pasting the gold document fixes the output, choose retrieval; if the model still misbehaves with perfect context, the deficit is in the weights and only fine-tuning (or better prompting) moves it.
(2) Update Latency And Provenance: RAG updates at index write speed and can return source spans for audit, whereas a fine-tuned model has no pointer back to evidence and its knowledge is frozen at the last training run.
(3) Cost Moves From Training To Serving: retrieval adds a lookup plus $k \cdot n_{chunk}$ prefill tokens on every query, while fine-tuning pays once up front and then serves short prompts, which matters under a tight token budget or a strict p99 latency SLO.
(4) Failure Modes Are Different: RAG fails by retrieval miss or by grounding on a plausible distractor chunk; fine-tuning fails by confidently asserting stale facts and by catastrophic forgetting of general ability.
(5) The Honest Answer Is Often Both: use RAG for the facts and a small LoRA adapter for format, jargon, and the skill of reading retrieved context (including ignoring irrelevant chunks), which is what retrieval-aware fine-tuning trains explicitly.

Flowchart: a failure measured on a held-out eval set fans out into three branches, knowledge gap, behavior gap, and both gaps, which route to RAG updated by reindexing, a LoRA fine-tune updated by retraining, and retrieval-aware fine-tuning that runs both pipelines, with per-query cost notes beneath each option

Figure 1: The decision starts from a measured failure, not from a technology preference: classify the gap first, then accept the update path and per-query cost that come with the chosen option.

The serving arithmetic is worth doing before committing. A 60-token question with $k = 5$ retrieved chunks of 400 tokens each becomes a 2060-token prefill, roughly 34 times the baseline prompt, plus 20 to 80 ms of vector search and reranking. That is usually cheaper than a training run, but it is a recurring per-request tax, and it is why teams with a small static corpus sometimes skip retrieval entirely and rely on long context with prompt caching. Fine-tuning inverts the profile: one-off data curation and GPU hours, then thin prompts forever. A common production shape is retrieve → rerank → generate on a live index, with a LoRA adapter handling the output contract, so the two mechanisms address the two gaps independently instead of competing.

Mathematical Formulation:
$p(y \mid x) = \sum_{z \in \mathcal{Z}_k} p_{\eta}(z \mid x) p_{\theta}(y \mid x, z)$
$\theta^{*} = \arg\min_{\theta} \mathcal{L}(\theta; \mathcal{D}_{ft})$
$n_{prefill} = n_{q} + k \cdot n_{chunk}$
$t_{total} = t_{retrieve} + t_{prefill} + t_{decode}$

Where:

$y$ is the generated answer and $x$ the user query; the first line is the RAG marginalization, where new knowledge enters through the retrieved set rather than through the weights.
$z$ is a retrieved passage from the top- $k$ set $\mathcal{Z}_k$ , scored by the retriever $p_{\eta}$ , and $p_{\theta}$ is the generator; only $\mathcal{Z}_k$ changes when you reindex.
$\theta^{*}$ is the fine-tuned parameter set obtained by minimizing loss $\mathcal{L}$ on the curated set $\mathcal{D}_{ft}$ ; with LoRA only a low-rank delta is trained, so $\theta$ itself stays frozen.
$n_{q}$ is the question length, $n_{chunk}$ the tokens per chunk, and $n_{prefill}$ the per-query context the RAG path must pay for; fine-tuning leaves $n_{prefill} = n_{q}$ .
$t_{retrieve}$ , $t_{prefill}$ and $t_{decode}$ are the latency terms; a hard SLO such as $t_{total} \leq 300$ ms with $k = 20$ chunks is where retrieval budgets usually break.

Line chart of fact-level answer accuracy over 365 days after a training cutoff: a fine-tuned checkpoint with no retraining decays from 0.88 to about 0.30, quarterly retraining produces a sawtooth that recovers to 0.88 every 91 days, and RAG over a live index stays flat near 0.86

Figure 2: Illustrative fact churn of about 15 accuracy points per quarter: a frozen fine-tuned checkpoint decays, quarterly retraining buys a sawtooth that is stale by construction between runs, and a live index holds accuracy flat with no training at all.

Dimension	RAG	Fine-Tuning	Both
Best for	Changing, private, or high-volume facts that need citations	Output format, tone, domain jargon, tool-call syntax, latency cuts	Grounded answers in a strict contract, with noisy retrieval
Update path	Reindex the changed chunk, minutes, no GPU	New data, training run, regression eval, redeploy	Reindex for facts, retrain only when behavior shifts
Provenance	Source spans returned with the answer	None; knowledge is diffuse in weights	Same as RAG, with better citation discipline
Per-query cost	Retrieval latency plus $k \cdot n_{chunk}$ prefill tokens	Short prompt only; cheapest at high QPS	Same as RAG, plus adapter serving
Main failure mode	Retrieval miss, distractor chunks, lost-in-the-middle	Stale confident facts, catastrophic forgetting, overfit style	Two systems to version, debug, and evaluate together
Build effort	Chunking, embeddings, index, reranker, eval set	Labeled pairs, hyperparameters, GPU time, eval harness	Highest, so justify it with measured gains over RAG alone

August 15, 2026

DL0083 LLM Hallucination and Reduction

What is hallucination in LLMs, and what techniques reduce it?

Answer

A hallucination is fluent, confident model output that is not supported by the provided source or by verifiable fact. The standard split is intrinsic (the output contradicts a document that was placed in the context) versus extrinsic (the output adds unsupported content, such as an invented citation, API argument, or date). Hallucination is structural rather than a bug in any single checkpoint: next-token maximum likelihood rewards plausible continuations, not true ones, parametric memory of rare facts is lossy, and benchmarks scored on plain accuracy give zero credit for saying “I don’t know,” so a guess strictly dominates an abstention during both evaluation and preference tuning. Reduction therefore works on three fronts at once: put the evidence in the context (retrieval grounding with citations), check the produced claims against that evidence (claim-level verification), and change the objective so the model is allowed to abstain when its confidence is below the scoring threshold.

(1) Grounding Beats Recall: retrieving the passage and requiring an inline citation converts a memory problem into a reading-comprehension problem, which is what cuts unsupported claims the most for factual queries.
(2) Verify At Claim Granularity: decompose the answer into atomic claims and run an entailment (NLI) check of each claim against its cited span; a single unsupported claim in an otherwise correct paragraph is exactly what whole-answer scoring misses.
(3) Uncertainty Signals: sampling several answers and measuring disagreement over meanings rather than token strings (semantic entropy, SelfCheckGPT-style consistency checks) flags confabulation without any external corpus.
(4) Calibrated Abstention: a fixed answer-or-abstain threshold derived from the scoring rule trades coverage for precision, and is the only technique that removes hallucinations the retriever never had evidence for.
(5) Decoding And Tuning Help At The Margin: contrastive decoding against earlier layers or against the ungrounded prior, plus fine-tuning that rewards “insufficient evidence” responses, reduce sycophancy and copy-drift but do not manufacture missing knowledge.

Grouped bar chart comparing hallucination rate and answer coverage for five configurations: closed-book baseline at 42 percent hallucination and 100 percent coverage, chain of thought with self-consistency at 31 and 100, RAG top-5 at 18 and 98, RAG plus claim verification at 11 and 95, and RAG plus verification plus abstention at 4 percent hallucination but only 78 percent coverage

Figure 1: Illustrative long-tail QA sweep: grounding and verification push the hallucination rate from 42% to 11% while coverage stays near 95%, and only abstention reaches 4%, at the cost of dropping coverage to 78%.

In production the pieces are wired as one loop rather than as independent tricks: retrieve → generate with citations → extract claims → entail each claim against its cited span, and route anything unsupported to a second retrieval pass or to an explicit refusal. Two details decide whether the loop actually helps. First, the verifier must judge support, not plausibility, so an NLI model or a judge restricted to the cited span is required; asking the same generator “is this correct?” mostly reproduces its original error. Second, a retrieval miss must be visible: if the top-k passages are off-topic and the prompt still demands an answer, the model quietly falls back to parametric memory and attaches a real-looking citation to a fabricated statement, which is the worst failure mode because it survives casual review. Systems that expose this path well (Google’s Vertex AI grounding returns per-claim support scores alongside the answer) make the unsupported fraction a monitorable metric instead of an anecdote.

Left to right pipeline of six boxes: user query, retrieve top-k passages, generate answer with citations, split into atomic claims, NLI check versus cited passage, return answer plus evidence, with a dashed feedback path from the NLI check back to the retriever labelled unsupported claim, re-retrieve or abstain

Figure 2: The verification loop: every atomic claim is entailment-checked against the span it cites, and unsupported claims trigger re-retrieval or abstention instead of being returned with a decorative citation.

Mathematical Formulation:
$\mathbb{E}[S] = p - c\,(1 - p)$
$p \geq \frac{c}{1 + c}$
$p(C_k \mid x) = \sum_{s \in C_k} p(s \mid x)$
$H_{sem}(x) = -\sum_{k=1}^{K} p(C_k \mid x)\,\log p(C_k \mid x)$

Where:

$\mathbb{E}[S]$ is the expected score of answering under a rule that pays $+1$ for correct, $-c$ for wrong, and $0$ for abstaining; $p$ is the model’s probability that its answer is correct.
The second line is the abstention threshold: answer only when $p$ clears it. Plain accuracy scoring is the case $c = 0$ , where the threshold collapses to $0$ and guessing is never penalized, which is the incentive that trains hallucination into the model.
$x$ is the prompt and $s$ a sampled generation; $C_k$ is a semantic equivalence class of generations grouped by bidirectional entailment, with $k \in \{1,\ldots,K\}$ indexing the $K$ distinct meanings observed.
$H_{sem}(x)$ is the semantic entropy: it is near $0$ when all samples paraphrase one meaning and large when the model spreads mass over mutually contradictory meanings, so a high value is a practical confabulation flag even when token-level entropy is low.

Technique	What It Actually Fixes	Cost	When to Reach for It
Retrieval grounding with citations	Missing or stale parametric knowledge; long-tail entities	Index build plus retrieval latency and a much longer prompt	Default for any factual, enterprise, or time-sensitive query
Claim extraction plus NLI verifier	Answers that cite a real source but overstate what it says	One extra model call per claim; adds hundreds of ms	Regulated or high-stakes surfaces where citations are shown to users
Sampling self-checks and semantic entropy	Confabulation on questions with no retrievable source	5x to 20x tokens; misses confident systematic errors	Offline audits, dataset cleaning, and routing to human review
Calibrated abstention and refusal tuning	Guessing when evidence is absent; sycophantic agreement	Lower coverage and more “I cannot verify this” responses	When a wrong answer costs far more than a missing one
Contrastive decoding (DoLa, context-aware)	Drift back to the parametric prior while a document is in context	Cheap, but needs logit access and per-model tuning	Summarization and closed-context QA on self-hosted models

August 15, 2026

DL0082 LLM Temperature

What is LLM temperature?

Answer

Temperature is a single positive scalar that divides the model’s logits before the softmax at every decoding step, so it reshapes the next-token distribution without touching the model weights. At $T = 1$ you sample from exactly the distribution the model was trained to produce; values below 1 sharpen that distribution toward the highest-scoring token, and values above 1 flatten it toward uniform over the vocabulary. The transformation is monotone in the logits, so temperature never changes which token is ranked first, only how much probability the gaps between logits are worth. The practical effect is a variance knob: low temperature gives repetitive but stable text, high temperature gives diverse text with a much fatter tail of implausible tokens. Because it is applied independently at every step of an autoregressive loop, a small per-token change in tail mass compounds across a long generation.

(1) Logit Rescaling, Not Reranking: dividing all logits by the same $T$ preserves their order, so the argmax token is invariant and only the sampling probabilities move.
(2) The Two Limits: as $T$ approaches 0 sampling collapses to greedy decoding, and as $T$ grows the distribution approaches uniform; frameworks special-case $T = 0$ as argmax because the division itself is undefined.
(3) Tails Gain Disproportionately: temperature is a power transform of the base distribution, so unlikely tokens change by the largest relative factor; with logits $(4, 3, 2, 1)$ the top token falls from 0.644 to 0.455 at $T = 2$ while the weakest rises from 0.032 to 0.102.
(4) Compounding Across Steps: the model conditions on its own samples, so one low-quality token drawn from the inflated tail can steer every subsequent token, which is why high temperature degrades long generations faster than short ones.
(5) Not A Truthfulness Knob: lowering $T$ reduces variance, not error; if the mode is wrong, greedy decoding returns the wrong answer with total confidence.

Grouped bar chart of next-token probabilities for four tokens with logits 4.0, 3.0, 2.0 and 1.0 at temperatures 0.5, 1.0 and 2.0, showing the top token falling from 0.865 to 0.455 and the weakest token rising from 0.002 to 0.102

Figure 1: The same four logits under three temperatures. Entropy grows from 0.66 bits at $T = 0.5$ to 1.80 bits at $T = 2$ , and the weakest token gains roughly 47x probability across that range while the top token loses less than half.

Temperature never acts alone in a real decoder. The usual pipeline is logits → temperature → top-k or top-p truncation → renormalize → sample, and the ordering matters: because temperature runs first, raising it inflates the tail that nucleus sampling then has to cut, so the same $p = 0.95$ threshold admits a larger candidate set than it did at $T = 1$ . That is why “high temperature plus top-p” is not a safety net, and why many production defaults pair a moderate temperature with a fixed truncation instead of pushing either one hard. A separate but easily confused use of the same formula is temperature scaling for calibration, where a single $T$ is fit on held-out data to correct an overconfident classifier; there the goal is matching predicted confidence to observed accuracy, not generating diverse text, and the fitted value is a property of the model rather than a user-facing creativity dial.

Mathematical Formulation:
$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^{V} \exp(z_j/T)}$
$p_i(T) \propto p_i(1)^{1/T}$
$i^{*} = \arg\max_j z_j$
$\lim_{T \to 0^{+}} p_{i^{*}}(T) = 1$
$H(T) = -\sum_{i=1}^{V} p_i(T) \log_2 p_i(T)$

Where:

$p_i(T)$ is the sampling probability of vocabulary token $i$ at temperature $T$ , and $z_i$ is that token’s raw logit from the final linear layer.
$V$ is the vocabulary size and $j$ indexes every candidate in the softmax denominator, so all probabilities are renormalized after rescaling.
$T > 0$ is the temperature: values below 1 sharpen the distribution, values above 1 flatten it, and $T = 1$ is an identity operation.
The second line shows temperature is a power transform with exponent $1/T$ applied to the base distribution, which is the compact reason tail tokens move by the largest relative factor.
$i^{*}$ is the argmax token, and the fourth line is the greedy limit; implementations therefore branch to argmax at $T = 0$ instead of evaluating the ratio.
$H(T)$ is the entropy of the next-token distribution in bits; it increases monotonically with $T$ and is bounded above by $\log_2 V$ .

Two panels versus temperature from 0.05 to 5: left panel shows entropy in bits rising monotonically from near 0 toward the 2-bit uniform limit and passing 1.37 bits at T equals 1; right panel shows top-token probability falling from 1 to about 0.33 while the mass of the two weakest tokens rises from near 0 to about 0.40

Figure 2: Sweeping $T$ over the same four logits. Entropy is monotonically increasing and saturates at $\log_2 4 = 2$ bits, while the tail mass of the two weakest tokens more than doubles between $T = 1$ and $T = 2$ .

Knob	Temperature	Top-k	Top-p (Nucleus)
What it changes	Rescales all logits; every token keeps nonzero probability	Keeps the k highest-scoring tokens, zeroes the rest	Keeps the smallest set whose mass reaches p, zeroes the rest
Adapts to context?	No, the same scalar at every step regardless of confidence	No, a fixed candidate count	Yes, the set shrinks when the model is confident
Typical failure mode	Too low: loops and boilerplate. Too high: off-topic or invented tokens	Fixed k is too tight on flat distributions and too loose on peaked ones	A high temperature inflates the tail, so the same p admits far more junk
When to reach for it	Global variance control, and generating diverse samples for self-consistency voting	Cheap hard cap on the candidate set, useful as a safety floor	Default truncation for open-ended text, paired with a modest temperature

August 15, 2026

DL0062 LSTM
What is an LSTM (Long Short-Term Memory) network? How does it address vanishing gradients?
Answer
A Long Short-Term Memory network is a gated recurrent neural network designed to carry useful information across many sequence steps. Each cell maintains a cell state and uses forget, input, and output gates to control what is retained, written, and exposed. Its additive cell-state update creates a shorter gradient path than repeatedly multiplying through a vanilla RNN’s nonlinear state transition, so gradients can remain useful when forget gates stay near one. LSTMs mitigate vanishing gradients rather than eliminating them: saturated gates, long products of forget factors, and poor optimization can still weaken learning.
(1) Two Recurrent States: The cell state $c_t$ is the long-term memory path, while the hidden state $h_t$ is the exposed representation used by the next step and downstream layers.
(2) Gated Update: The forget gate scales old memory, the input gate controls a candidate update, and the output gate selects how much of the updated memory becomes visible.
(3) Gradient Preservation: The derivative along the direct cell-state path contains products of forget gates instead of repeated full recurrent Jacobians; values near one preserve gradient flow.

Figure 1: One LSTM timestep: gated reads and writes surround an additive cell-state highway, while the output gate produces the hidden state.
Mathematical Formulation:
$f_t=\sigma\!\left(W_f[x_t,h_{t-1}]+b_f\right)$
$i_t=\sigma\!\left(W_i[x_t,h_{t-1}]+b_i\right)$
$\tilde c_t=\tanh\!\left(W_c[x_t,h_{t-1}]+b_c\right)$
$c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$
$o_t=\sigma\!\left(W_o[x_t,h_{t-1}]+b_o\right)$
$h_t=o_t\odot\tanh(c_t)$
Where:
$t$ is the timestep; $x_t$ is the current input, and $h_{t-1},c_{t-1}$ are the previous hidden and cell states.
$f_t,i_t,o_t\in(0,1)$ are element-wise forget, input, and output gates; $\tilde c_t$ is the candidate cell update.
$c_t$ is the updated long-term cell state and $h_t$ is the exposed hidden state.
$W_f,W_i,W_c,W_o$ and $b_f,b_i,b_c,b_o$ are learned affine parameters; $[x_t,h_{t-1}]$ denotes concatenation.
$\sigma$ is sigmoid, $\tanh$ is hyperbolic tangent, and $\odot$ is element-wise multiplication; the direct derivative includes $\partial c_t/\partial c_{t-1}=f_t$ .

Figure 2: Why LSTMs mitigate vanishing gradients: a vanilla RNN repeatedly multiplies full nonlinear Jacobians, whereas the LSTM provides a gated direct memory path.
Login to view more content
July 20, 2026

DL0049 Weight Init

Why is “weight initialization” important in deep neural networks?

Answer

Initialization decides whether signals survive a deep network. If weights are too small, activations and gradients shrink toward zero layer by layer (vanishing); too large, and they blow up (exploding) or saturate sigmoid/tanh into zero-gradient plateaus. Proper schemes like Xavier/Glorot and He scale the variance to the layer’s fan-in/fan-out so activations keep unit-scale statistics, while randomness breaks symmetry so neurons learn different features.

(1) Prevents Vanishing/Exploding Signals: Keeping activation variance constant across layers keeps gradients at a usable scale during backprop.
(2) Breaks Symmetry: Identical initial weights make neurons identical forever; random init gives each a distinct feature to learn.
(3) Matches the Activation: Xavier suits symmetric activations (tanh/sigmoid); He doubles the variance for ReLU, which zeroes half its inputs.

Mathematical Formulation:
$\text{Xavier:}\quad W \sim \mathcal{N}\!\left(0,\; \frac{2}{n_{\text{in}} + n_{\text{out}}}\right) \;\; \text{or} \;\; \mathcal{U}\!\left(-\sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}},\; \sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}\right)$
$\text{He:}\quad W \sim \mathcal{N}\!\left(0,\; \frac{2}{n_{\text{in}}}\right) \;\; \text{or} \;\; \mathcal{U}\!\left(-\sqrt{\frac{6}{n_{\text{in}}}},\; \sqrt{\frac{6}{n_{\text{in}}}}\right)$

Where:

$n_{\text{in}}$ and $n_{\text{out}}$ are the layer’s input and output unit counts (fan-in / fan-out).
Xavier balances both directions for symmetric activations; He drops $n_{\text{out}}$ and doubles variance because ReLU discards half the signal.

Three stacked histogram panels of post-ReLU activation distributions at layers 3 to 6, showing plain random init collapsing to a spike at zero, Xavier shrinking toward zero with depth, and He init keeping a stable spread.

Figure 1: Post-ReLU activations by depth: plain random init collapses and Xavier fades, while He init keeps a healthy spread even at layer 6.

Activation / Architecture	Recommended Init	Why
ReLU (CNNs, ResNet)	He / Kaiming	Doubles variance to compensate for ~50% zeros from ReLU
Tanh / Sigmoid (MLPs)	Xavier / Glorot	Balances fan-in and fan-out for symmetric activations
GELU (BERT-scale Transformers)	Truncated normal (σ ≈ 0.02)	LayerNorm + residuals already stabilize; gentle init suffices
Very deep LLMs (GPT, LLaMA)	Scaled normal (DeepNorm-style)	Residual-branch scaling stops signal growth across 100+ layers

Table 1: Init choice is activation- and architecture-dependent: there is no single best scheme.

October 28, 2025

DL0048 Adam Optimizer
Can you explain how the Adam optimizer works?
Answer
Adam (Adaptive Moment Estimation) combines momentum and RMSprop: it keeps an exponentially decaying average of the gradient (first moment, the direction) and of the squared gradient (second moment, the scale), then divides the former by the square root of the latter. The result is a per-parameter adaptive learning rate: large steps for parameters with small, consistent gradients, small steps for noisy or steep ones, plus bias correction that fixes the zero-initialization of both averages in early steps.
(1) First Moment (Momentum): $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t$ smooths the gradient direction over time.
(2) Second Moment (RMSprop): $v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$ tracks the gradient magnitude for per-parameter scaling.
(3) Bias Correction + Update: $\hat{m}_t = m_t/(1-\beta_1^t)$ , $\hat{v}_t = v_t/(1-\beta_2^t)$ remove the zero-init bias before the normalized step.
Mathematical Formulation:
$\theta_t = \theta_{t-1} - \alpha\, \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$
Where:
$\theta_t$ are the model parameters at step $t$ , and $g_t$ is the gradient of the loss at that step.
$\alpha$ is the learning rate (default $0.001$ ); $\epsilon \approx 10^{-8}$ prevents division by zero.
$\beta_1 = 0.9$ and $\beta_2 = 0.999$ control the decay of the first and second moment averages (paper defaults).

Figure 1: Adam on a quadratic bowl: adaptive per-parameter steps converge smoothly to the minimum without the zig-zag of plain SGD.
Intuition for the Division: $\hat{m}_t / \sqrt{\hat{v}_t}$ is roughly a signal-to-noise ratio: parameters whose gradients are large but inconsistent (high $v$ ) get small updates, while parameters with small but persistent gradients get amplified. This is what makes Adam robust to ill-scaled features and sparse gradients.
Login to view more content
October 27, 2025
DL0046 Focal Loss
What is focal loss, and why does it help with class imbalance?
Answer
Focal loss augments cross-entropy with a modulating factor $(1 - p_t)^\gamma$ that shrinks the loss of easy, well-classified examples toward zero, so the vast number of easy negatives in imbalanced data stops dominating the gradient. Training then concentrates on hard, misclassified, or minority-class examples, which is where the useful learning signal actually lives.
(1) Modulating Factor: $(1 - p_t)^\gamma$ is near 0 when the model is confident and correct, and near 1 when it is wrong, so easy examples are automatically down-weighted.
(2) Class Imbalance Effect: In detection-style problems with ~100k easy negatives per positive, plain cross-entropy is swamped by their accumulated gradient; focal loss suppresses it.
(3) Optional Balancing: A per-class weight $\alpha_t$ can additionally correct the base-rate skew of the classes themselves.
Mathematical Formulation:
$\mathrm{FL}(p_t) = -\alpha_t\, (1 - p_t)^\gamma \log(p_t)$
Where:
$p_t$ is the model’s predicted probability for the ground-truth class.
$\gamma \geq 0$ is the focusing parameter: larger values suppress easy examples harder; $\gamma = 0$ with $\alpha_t = 1$ recovers plain cross-entropy.
$\alpha_t \in (0,1)$ is an optional per-class balancing weight for class $t$ .

Figure 1: Focal loss curves for several $\gamma$ : as $\gamma$ grows, the loss of confident correct predictions (high $p_t$ ) collapses toward zero while hard examples keep full weight.
Scenario $p_t$ (True-Class Prob) Factor $(1-p_t)^\gamma$ Effect on Loss
Easy example High Low → 0 Down-weighted to near zero
Hard example Low High → 1 Keeps full learning signal
Table 1: How the modulating factor behaves: easy examples fade out, hard examples dominate the gradient.
Typical Settings: The RetinaNet paper found $\gamma = 2$ with $\alpha = 0.25$ (for the positive class) works well; the useful range is usually $\gamma \in [1, 3]$ ; too large over-focuses on noisy or mislabeled outliers.
Login to view more content
October 27, 2025
DL0043 KV Cache
What is KV Cache in transformers, and why is it useful during inference?
Answer
The KV cache stores the key and value vectors of all previously generated tokens so that autoregressive decoding never recomputes them. At each step the model computes Q, K, V only for the new token, appends K and V to the cache, and attends over the cached history, turning each decoding step from an $O(n^2 d)$ full recompute into an $O(n d)$ lookup, at the price of $O(n)$ growing memory.
(1) What Is Cached: Per layer, the projected keys and values $K_{1:t}, V_{1:t}$ of every past token; they never change once computed, so recomputing them is pure waste.
(2) Why It Helps: With causal masking, token $t$ attends to exactly $K_{1:t}, V_{1:t}$ ; caching reduces each step to one new query against a read-only cache.
(3) The Cost: Memory grows linearly ( $2 \cdot n \cdot d$ values per layer for K and V, $O(L \cdot n \cdot d)$ across $L$ layers) and becomes the inference bottleneck for long contexts and large batches.
Mathematical Formulation:
$\mathrm{Attention}(q_t, K_{1:t}, V_{1:t}) = \mathrm{softmax}\!\left(\frac{q_t K_{1:t}^{\top}}{\sqrt{d_k}}\right) V_{1:t}$
$\mathrm{Memory}_{\text{cache}} = 2 \cdot L \cdot n \cdot d_{\text{model}} \;\; \text{values}$
Where:
$q_t$ is the query of the current token $t$ ; $K_{1:t}, V_{1:t}$ are the cached keys and values for tokens $1 \ldots t$ .
$d_k$ is the per-head key dimension, $d_{\text{model}}$ the model width, and $L$ the number of layers; the factor 2 counts K and V buffers separately.

Figure 1: Without cache every step recomputes the full $Q, K, V$ ; with cache only the new token’s query is computed while cached K/V (purple) supply the history.
Practical Note: During training the full sequence is processed in parallel so the cache does not apply; it is an inference-only optimization, and its size is why long-context serving is memory-bound rather than FLOP-bound.
Login to view more content
September 14, 2025

Scenario	$p_t$ (True-Class Prob)	Factor $(1-p_t)^\gamma$	Effect on Loss
Easy example	High	Low → 0	Down-weighted to near zero
Hard example	Low	High → 1	Keeps full learning signal