← Roadmap

Reflexion: Language Agents with Verbal Reinforcement Learning

Shinn, Cassano, Berman, Gopinath, Narasimhan, Yao · 2023

① Intuition

Traditional reinforcement learning improves an agent by updating its neural network weights via gradients: the policy receives a reward signal, backpropagates through millions of parameters, and nudges its behavior for the next episode. This works beautifully when you own the model. But what if your agent is a black-box LLM accessed through an API — or any frozen model where gradient-based updates are impossible or prohibitively expensive? The weights are off limits. You cannot retrain a 100B-parameter model after every maze run.

Reflexion’s answer is verbal reinforcement: keep the weights frozen and instead let the agent update its behavior through language. After a failed trial, the agent — or a separate reflection module — writes a natural-language post-mortem (“the wall is at (2,2), I should navigate around it rather than through it”) and stores it in an episodic memory buffer. The next trial begins with this memory prepended to the prompt, effectively conditioning the policy on its own accumulated hindsight. No gradients. No weight updates. The agent improves by reading its own notes.

Before using the widget: imagine an agent navigating a grid with a hidden wall. On trial 1 it fails. On trial 2 it reads its own reflection ('the wall is at (2,2)'). Do you expect trial 2 to succeed? Why might it still fail, even with a perfect reflection?

Interact with the widget: step through 5 trials using “Next trial”. Start with “action only” to see what the agent did. Switch to “action + reward” to see the outcome. Finally switch to “action + reward + reflection” to read the verbal memory that gets appended before the next trial. Notice how the reflections accumulate and how the success rate changes after trial 3.

Trial 1 of 5

Action: Down, Right, Down, Right, Down, Right

0 of 1 trial reached goal

After playing with the widget: trial 5 in this demo is actually longer (path length 7 vs. 6) and the agent reflects on preferring shorter paths. What does this suggest about the limits of verbal reinforcement as the memory buffer grows?

Retrieval check

Reflexion updates the agent's neural network weights after each failed trial using the verbal reflection as a training signal. True or false?
In Reflexion, what is stored in the episodic memory buffer M?

Answer all 2 correctly to mark this stage mastered.

② Math

Reflexion formalizes three interacting modules. The Actor is the policy pi — an LLM that produces actions a_t from the current state s_t and the memory buffer M. The Evaluator computes a reward r_t from the completed trajectory — this can be binary (success/failure) or a scalar. The Self-Reflection module generates a natural-language feedback string from the trajectory and reward.

At trial t, the loop is:

τt=(s0, a1, s1, …, aT, sT)∼π(⋅∣s0, Mt−1)\tau_t = \bigl(s_0,\,a_1,\,s_1,\,\ldots,\,a_T,\,s_T\bigr) \sim \pi(\cdot \mid s_0,\, M_{t-1})

The actor unrolls a trajectory tau_t by sampling actions from the policy conditioned on the initial state and the current memory. Then:

rt=Evaluator(τt)r_t = \text{Evaluator}(\tau_t)ℓt=Reflector(τt,  rt)\ell_t = \text{Reflector}(\tau_t,\; r_t)Mt=Mt−1∪{ℓt}M_t = M_{t-1} \cup \{\ell_t\}

The memory grows by one reflection per trial. The policy at trial t+1 conditions on M_t. There is no explicit gradient anywhere in this loop — the only “learning” is the expansion of M.

Why is a verbal reflection richer than a scalar reward as a learning signal?
Show next line
rt∈R  (scalar)vs.ℓt∈NL(step,cause,fix)r_t \in \mathbb{R} \;\text{(scalar)}\quad\text{vs.}\quad \ell_t \in \text{NL}(\text{step}, \text{cause}, \text{fix})
What architectural constraint limits the size of memory M, and what happens when that limit is reached?
Show next line
∣Mt (tokens)∣≤context window⇒summarize or drop old ℓi|M_t \text{ (tokens)}| \leq \text{context window} \Rightarrow \text{summarize or drop old } \ell_i

Retrieval check

In the Reflexion update equation M_t = M_{t-1} union {l_t}, what triggers the generation of l_t?
The Reflexion Actor samples actions from pi(· | s, M). What changes between trial t and trial t+1?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the five steps of the Reflexion agent loop in the correct order:

  1. Run trajectory tau_t by sampling actions from Actor(pi, s_0, M)
  2. Repeat until max trials reached or task succeeds
  3. Evaluate trajectory: r_t = Evaluator(tau_t)
  4. Initialize memory M as empty list; reset environment to initial state s_0
  5. Generate verbal reflection: l_t = Reflector(tau_t, r_t); append l_t to M

Fill in the blank — the call that generates and stores a verbal reflection:

def reflexion_step(trajectory, reward, memory):
    reflection = _______
    memory.append(reflection)
    return memory

Retrieval check

In the pseudo code loop, the memory M is passed to the Actor at the start of each trial. Why must M be passed at the beginning of the trial, not at the end?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement a minimal Reflexion memory loop in Python. Given a task string, a stub function run_trial(task, memory) that returns (reward, trajectory), and a stub function reflect(trajectory, reward) that returns a string, write reflexion_loop to run n_trials trials, accumulating reflections in a list and returning the final memory.

Retrieval check

In the reflexion_loop implementation, memory is passed to run_trial on every iteration. What does this test?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Reflexion: Language Agents with Verbal Reinforcement Learning (Shinn et al. 2023).

Open in new tab · Download

Suggested reading order: §1 (Introduction — motivation for verbal RL and contrast with gradient-based RL) → §3 (Reflexion — formal description of the three modules and the memory loop) → §4 (Experiments — AlfWorld, HotpotQA, and HumanEval benchmarks).