Turn task feedback into inspectable memory changes. Separate record updates, retrieval policy changes, and model training, then test on subsequent tasks.
A feedback loop: select useful records from task experience, retrieve and use them in a later task, then use the outcome to improve the memory policy.
After a failed task, a note saying “be more careful next time” gives an agent little guidance. A useful update identifies what failed, the evidence for that diagnosis, and the conditions under which the next attempt should change.
A practical feedback loop records which memories were used, locates the failure, changes the relevant record or policy, and checks later tasks.
Distinguish what is being improved
| Object | Change | Evidence needed |
|---|---|---|
| Memory content | Correct an outdated rule or add conditions | Before-and-after records and supporting sources |
| Memory policy | Change when to write, retrieve, or rank records | The policy change and a controlled task comparison |
| Model parameters | Train the model itself | Training objective, data, parameter updates, and separate evaluation |
External memory can change an agent's behavior without updating model weights. Reflexion investigates converting task feedback into verbal reflections stored in episodic memory for later attempts.
Locate the failure before changing the policy
A missing lesson may never have been written. It may have been stored but not retrieved. A correct record may have been retrieved and ignored, or a lesson for an older software version may have been applied to a new one.
Trace the same record through the process: does the original evidence exist, did the summary preserve its conditions, was it retrieved, and did it influence the action? Final success or failure alone cannot tell you whether to change writing, retrieval, or the agent's use of evidence.
Turn a failure into a usable record
Suppose a repository has several workspaces, but its memory presents one test command as universal. Running it from the root fails. Running the appropriate command inside the web workspace succeeds.
Saving “change directory first” is still too vague. Record the repository and workspace, the failed command and location, the successful command, the version checked, and the logs. On the next relevant task, this record should guide the agent to identify the workspace and confirm the command in current configuration.
If the project later adds a unified root test command, update the lesson's scope. The old failure can remain useful history while current recommendations point to the new entry point.
A successful task does not validate every retrieved memory
An agent may retrieve several records but solve the task mainly from current code. Passing tests does not establish that every retrieved record deserves a higher weight. A failure may also come from the environment, network, or test itself.
Keep separate observations: what was retrieved, which advice was followed, what happened, and what a person corrected. Leave records unchanged when their contribution cannot be identified. Otherwise, an incidental success can become a misleading permanent rule.
Check the update on later tasks
First rerun the original failure to verify the correction. Then use related tasks that did not contribute to the update. Another change in the same workspace should find the right test entry point; a task in a different workspace should not inherit it incorrectly.
Hold the model, tools, and task budget constant. Compare completion, repeated errors, manual corrections, and total cost before and after the update. Keep an update when it helps within its intended scope without misleading tasks outside it.
References https://arxiv.org/abs/2303.11366v4
