① Intuition
By 2022 large language models like GPT-3 had shown impressive in-context learning on many tasks. But on multi-step arithmetic and commonsense reasoning benchmarks, even a 540B-parameter model would fail by jumping straight to a wrong answer. Fine-tuning helped little: the bottleneck was not knowledge but the inability to decompose hard problems into smaller steps. Simply scaling up further did not fix this — at 540B params, GSM8K accuracy with standard prompting sat around 18%.
Chain-of-Thought (CoT) prompting changes the prompt format rather than the model weights. Instead of showing the model (question, answer) pairs in the few-shot prompt, you show (question, reasoning steps, answer) triplets. The model then imitates this format on the query — first writing a chain of reasoning, then writing the final answer. That single format change pushed GSM8K accuracy from ~18% to ~57% on PaLM 540B. Remarkably, it only helps at roughly 60B+ parameters; smaller models produce incoherent chains that hurt accuracy.
Before using the widget: when you toggle Chain-of-Thought on for the apples problem, what do you expect to see differently in the output? Will the answer change, and if so, why?
With CoT on, the model first writes out each arithmetic step before giving the final answer. The direct-answer path skips the decomposition and lands on the wrong number. The reasoning path forces each sub-computation to be explicit and checkable — errors in one step are easier to catch because they are visible. This is why CoT helps: it converts a single hard compositional inference into a sequence of simpler local ones.
Interact with the widget: select a problem from the dropdown, then toggle Chain-of-Thought on and off. Observe that the direct answer is wrong while the reasoning chain produces the correct result. Try all three problems to see that the pattern holds across different arithmetic structures.
After playing with the widget: if CoT prompting works by imitating the exemplar format, what would happen if you added 'Let's think step by step.' to the prompt but without any reasoning exemplars? Would it still help?
Yes — this is zero-shot CoT (Kojima et al., 2022). Appending the phrase “Let’s think step by step.” to the query, without any few-shot examples at all, elicits a reasoning chain from the model. It achieves roughly 3× the accuracy of standard zero-shot prompting on arithmetic benchmarks. The phrase acts as a trigger that activates the model’s latent chain-of-thought generation capability — capability that only emerges at large scale.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
Standard few-shot prompting maximizes the probability of the answer directly:
where x is the question and a is the answer. Chain-of-thought prompting instead factors generation through an intermediate reasoning chain r:
In practice the model generates a single high-probability chain r* (greedy or beam search), then extracts the answer:
The reasoning chain r* decomposes the hard compositional problem into a sequence of simpler local steps, each of which is individually more tractable for the model. In the marginal p(a | r*, x), the answer is now conditioned on a context that has already resolved the intermediate quantities.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the four steps of chain-of-thought inference in the correct order:
- Append the query question to the prompt
- Construct k-shot prompt with (question, reasoning, answer) exemplars
- Extract the final numeric or text answer from the generation
- LM generates the reasoning chain, then the final answer
Fill in the blank — the key suffix that triggers zero-shot chain-of-thought:
def zero_shot_cot(query, model):
prompt = query + "\n" + _______
return model.generate(prompt)Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement a few-shot CoT prompt builder. Given a list of examples (each a dict with "question", "reasoning", and "answer" keys) and a query string, return the full prompt string. Each exemplar should be formatted as:
Q: <question>
A: <reasoning> The answer is <answer>.followed by Q: <query>\nA: at the end to elicit the model’s completion.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §1 (Introduction — the GSM8K motivation and the core claim) → §3 (Chain-of-Thought Prompting — the method, exemplar format, and zero-shot CoT) → §4 (Arithmetic Reasoning — benchmark results, scaling curve, and emergent behavior at large model size).