← Roadmap

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le, Zhou · 2022

① Intuition

By 2022 large language models like GPT-3 had shown impressive in-context learning on many tasks. But on multi-step arithmetic and commonsense reasoning benchmarks, even a 540B-parameter model would fail by jumping straight to a wrong answer. Fine-tuning helped little: the bottleneck was not knowledge but the inability to decompose hard problems into smaller steps. Simply scaling up further did not fix this — at 540B params, GSM8K accuracy with standard prompting sat around 18%.

Chain-of-Thought (CoT) prompting changes the prompt format rather than the model weights. Instead of showing the model (question, answer) pairs in the few-shot prompt, you show (question, reasoning steps, answer) triplets. The model then imitates this format on the query — first writing a chain of reasoning, then writing the final answer. That single format change pushed GSM8K accuracy from ~18% to ~57% on PaLM 540B. Remarkably, it only helps at roughly 60B+ parameters; smaller models produce incoherent chains that hurt accuracy.

Before using the widget: when you toggle Chain-of-Thought on for the apples problem, what do you expect to see differently in the output? Will the answer change, and if so, why?

Interact with the widget: select a problem from the dropdown, then toggle Chain-of-Thought on and off. Observe that the direct answer is wrong while the reasoning chain produces the correct result. Try all three problems to see that the pattern holds across different arithmetic structures.

Emma has 3 baskets with 7 apples each. She gives away 5 apples. How many left?
Direct answer: 25

After playing with the widget: if CoT prompting works by imitating the exemplar format, what would happen if you added 'Let's think step by step.' to the prompt but without any reasoning exemplars? Would it still help?

Retrieval check

Chain-of-Thought prompting requires modifying model weights via fine-tuning to inject the reasoning capability. True or false?
CoT prompting was found to help across all model sizes. True or false?

Answer all 2 correctly to mark this stage mastered.

② Math

Standard few-shot prompting maximizes the probability of the answer directly:

pθ(a∣x)p_\theta(a \mid x)

where x is the question and a is the answer. Chain-of-thought prompting instead factors generation through an intermediate reasoning chain r:

pθ(a∣x)=∑rpθ(a∣r, x)  pθ(r∣x)p_\theta(a \mid x) = \sum_r p_\theta(a \mid r,\, x)\; p_\theta(r \mid x)

In practice the model generates a single high-probability chain r* (greedy or beam search), then extracts the answer:

a^=arg⁡max⁡a  pθ ⁣(a∣r∗,x),r∗=arg⁡max⁡r  pθ(r∣x)\hat{a} = \arg\max_a\; p_\theta\!\left(a \mid r^*, x\right), \quad r^* = \arg\max_r\; p_\theta(r \mid x)

The reasoning chain r* decomposes the hard compositional problem into a sequence of simpler local steps, each of which is individually more tractable for the model. In the marginal p(a | r*, x), the answer is now conditioned on a context that has already resolved the intermediate quantities.

Why does chain-of-thought prompting only help at large model scale (≥60B params) rather than all sizes?
Show next line
r∗ coherent⇒p(a∣r∗,x) high(requires scale)r^* \text{ coherent} \Rightarrow p(a \mid r^*, x) \text{ high} \quad \text{(requires scale)}
Zero-shot CoT appends 'Let's think step by step.' to the prompt without any few-shot examples. In the probability factorization above, what role does this phrase play?
Show next line
’Let’s think step by step’⇒p(r∣x) shifts toward structured chains\text{'Let's think step by step'} \Rightarrow p(r \mid x) \text{ shifts toward structured chains}

Retrieval check

In the chain-of-thought factorization, the answer probability is conditioned on the reasoning chain r*. What is the consequence if r* contains a factual error?
Standard few-shot prompting and CoT prompting both use k exemplars in the context. What is the key structural difference between the exemplars?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the four steps of chain-of-thought inference in the correct order:

  1. Append the query question to the prompt
  2. Construct k-shot prompt with (question, reasoning, answer) exemplars
  3. Extract the final numeric or text answer from the generation
  4. LM generates the reasoning chain, then the final answer

Fill in the blank — the key suffix that triggers zero-shot chain-of-thought:

def zero_shot_cot(query, model):
    prompt = query + "\n" + _______
    return model.generate(prompt)

Retrieval check

In the pseudo code above, the answer is extracted from the generation after the reasoning chain. Why is answer extraction a non-trivial step in practice?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement a few-shot CoT prompt builder. Given a list of examples (each a dict with "question", "reasoning", and "answer" keys) and a query string, return the full prompt string. Each exemplar should be formatted as:

Q: <question>
A: <reasoning> The answer is <answer>.

followed by Q: <query>\nA: at the end to elicit the model’s completion.

Retrieval check

In the prompt builder above, the final line ends with 'A:' and nothing else. Why is this specific ending important for eliciting a chain-of-thought response from the model?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al. 2022).

Open in new tab · Download

Suggested reading order: §1 (Introduction — the GSM8K motivation and the core claim) → §3 (Chain-of-Thought Prompting — the method, exemplar format, and zero-shot CoT) → §4 (Arithmetic Reasoning — benchmark results, scaling curve, and emergent behavior at large model size).