DL0114 Chain-of-Thought Prompting

Chain-of-Thought (CoT) prompting makes a language model emit intermediate reasoning steps before its final answer, either through few-shot exemplars whose demonstrations contain worked solutions (Wei et al., 2022) or through a zero-shot trigger phrase such as “Let’s think step by step” (Kojima et al., 2022). The gains are large on multi-step problems: on GSM8K, 8-shot CoT lifted PaLM 540B from 17.9% to 56.9%, and the same trigger took GPT-3 (text-davinci-002) from 10.4% to 40.7% with no weight updates. Two mechanisms explain why. The first is serial compute: a transformer with layers can apply only sequential operations before it must commit to the next token, so a task needing more sequential steps than the depth provides is simply not representable in one forward pass; every generated token reruns the whole stack, turning the context window into an external scratchpad that raises effective serial depth from to . Theory backs this up: constant-depth transformers with a polynomially long chain can simulate any polynomial-time computation, while the same model restricted to direct answers cannot. The second mechanism is factorized conditioning: instead of sampling one high-entropy jump from question to answer, the model decomposes the problem into a product of low-entropy conditionals, each conditioned on the already-written steps, which keeps it on the distribution of solution traces seen in pretraining. CoT is not universally free money: it is emergent with scale, and later meta-analysis shows the gains concentrate on math, symbolic, and logical tasks rather than on knowledge or commonsense retrieval.

(1) Two Elicitation Modes: few-shot CoT shows worked examples and controls format tightly, while zero-shot CoT appends one trigger sentence and costs almost no prompt tokens.
(2) Depth Becomes Time: each scratchpad token buys another full pass through the network, so the model trades latency and tokens for sequential computation it structurally lacked.
(3) Easier Conditionals: the chain rewrites one hard prediction as many easy ones, and each step reads all earlier steps as ordinary context.
(4) Emergent, Not Universal: below roughly 10B parameters, CoT often produces fluent but invalid chains and can score below direct prompting.
(5) Chains Are Not Guaranteed Faithful: models shift answers when a prompt carries a bias cue while the written rationale never mentions it, so a chain is an output artifact, not an audit trail.
(6) Cheap Ensembling On Top: because chains are stochastic, sampling of them and majority-voting the final answers (self-consistency) pushed PaLM 540B on GSM8K to 74.4%.

Figure 1: Direct prompting caps sequential computation at the network depth ; CoT reuses the same weights once per generated token, giving sequential steps and letting step read every earlier step as context.

The costs are as concrete as the benefits. Chains are typically 100 to 400 decode tokens on grade-school math, so a CoT query can cost an order of magnitude more decode passes and time-to-last-token than a direct answer, which matters for anything user-facing. Errors also propagate: the decoder never backtracks, so a slip in the first arithmetic step is carried through every later step, and accuracy behaves roughly like the product of per-step accuracies. Few-shot CoT is additionally prompt-sensitive, since exemplar choice, ordering, and even the formatting of the equations move accuracy by several points. These properties are exactly why the field moved from prompting toward verifying or training the chain rather than just requesting it.

Mathematical Formulation:

Where:

• is the prompt (question plus any exemplars), the final answer span, and the generated rationale, a latent variable the model writes into its own context.
• indexes chain tokens; is the chain length and the number of transformer layers.
• counts the sequential layer applications available before the answer is committed, which is the resource CoT actually adds.
• is the number of sampled chains in self-consistency, the answer extracted from sample , and the indicator used for majority voting.
• Required condition: the sum over is intractable, so plain CoT approximates it with a single greedy or sampled , and self-consistency approximates it with Monte Carlo samples at temperature .

Figure 2: CoT is an emergent ability: at 8B parameters the chains are fluent but wrong and buy nothing, while at 540B they roughly triple the GSM8K solve rate. Values are approximate figures reported for PaLM by Wei et al. (2022) and Wang et al. (2023).