① Intuition
Traditional reinforcement learning improves an agent by updating its neural network weights via gradients: the policy receives a reward signal, backpropagates through millions of parameters, and nudges its behavior for the next episode. This works beautifully when you own the model. But what if your agent is a black-box LLM accessed through an API — or any frozen model where gradient-based updates are impossible or prohibitively expensive? The weights are off limits. You cannot retrain a 100B-parameter model after every maze run.
Reflexion’s answer is verbal reinforcement: keep the weights frozen and instead let the agent update its behavior through language. After a failed trial, the agent — or a separate reflection module — writes a natural-language post-mortem (“the wall is at (2,2), I should navigate around it rather than through it”) and stores it in an episodic memory buffer. The next trial begins with this memory prepended to the prompt, effectively conditioning the policy on its own accumulated hindsight. No gradients. No weight updates. The agent improves by reading its own notes.
Before using the widget: imagine an agent navigating a grid with a hidden wall. On trial 1 it fails. On trial 2 it reads its own reflection ('the wall is at (2,2)'). Do you expect trial 2 to succeed? Why might it still fail, even with a perfect reflection?
The reflection gives the agent a crucial piece of information it lacked — the wall location — so trial 2 is more likely to avoid that specific obstacle. But verbal reinforcement is not guaranteed: the reflection must be correctly generated, the agent must attend to it and act on it, and the corrected path must not encounter new obstacles. Reflexion succeeds when the reflection accurately diagnoses the failure mode and the next trial’s task is within the agent’s in-context reasoning ability.
Interact with the widget: step through 5 trials using “Next trial”. Start with “action only” to see what the agent did. Switch to “action + reward” to see the outcome. Finally switch to “action + reward + reflection” to read the verbal memory that gets appended before the next trial. Notice how the reflections accumulate and how the success rate changes after trial 3.
Trial 1 of 5
0 of 1 trial reached goal
After playing with the widget: trial 5 in this demo is actually longer (path length 7 vs. 6) and the agent reflects on preferring shorter paths. What does this suggest about the limits of verbal reinforcement as the memory buffer grows?
The agent can only attend to a finite context window. As reflections accumulate — some about walls, some about path length, some potentially contradicting each other — the agent must weigh multiple constraints simultaneously. Older reflections may be pushed out of context or diluted by newer ones. Reflexion’s paper explicitly notes that a large memory buffer can hurt performance if older, still-relevant reflections are truncated. This is the verbal analogue of catastrophic forgetting: not in weights, but in context.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
Reflexion formalizes three interacting modules. The Actor is the policy pi — an LLM that produces actions a_t from the current state s_t and the memory buffer M. The Evaluator computes a reward r_t from the completed trajectory — this can be binary (success/failure) or a scalar. The Self-Reflection module generates a natural-language feedback string from the trajectory and reward.
At trial t, the loop is:
The actor unrolls a trajectory tau_t by sampling actions from the policy conditioned on the initial state and the current memory. Then:
The memory grows by one reflection per trial. The policy at trial t+1 conditions on M_t. There is no explicit gradient anywhere in this loop — the only “learning” is the expansion of M.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the five steps of the Reflexion agent loop in the correct order:
- Run trajectory tau_t by sampling actions from Actor(pi, s_0, M)
- Repeat until max trials reached or task succeeds
- Evaluate trajectory: r_t = Evaluator(tau_t)
- Initialize memory M as empty list; reset environment to initial state s_0
- Generate verbal reflection: l_t = Reflector(tau_t, r_t); append l_t to M
Fill in the blank — the call that generates and stores a verbal reflection:
def reflexion_step(trajectory, reward, memory):
reflection = _______
memory.append(reflection)
return memoryRetrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement a minimal Reflexion memory loop in Python. Given a task string, a stub function run_trial(task, memory) that returns (reward, trajectory), and a stub function reflect(trajectory, reward) that returns a string, write reflexion_loop to run n_trials trials, accumulating reflections in a list and returning the final memory.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §1 (Introduction — motivation for verbal RL and contrast with gradient-based RL) → §3 (Reflexion — formal description of the three modules and the memory loop) → §4 (Experiments — AlfWorld, HotpotQA, and HumanEval benchmarks).