How SWE-ContextBench evaluates experience reuse across related coding tasks through resolution accuracy, runtime, and token cost.
After fixing a problem in a module, can a coding agent use that experience to reduce repeated investigation and correctly solve a related task? SWE-ContextBench connects prior task experience with later problems to evaluate that question.
Version 3 of the paper by Jiayuan Zhu and colleagues constructs related tasks using GitHub issue and pull-request relationships and compares context reuse settings. It measures whether earlier experience improves subsequent problem solving.
Project information and task experience are different resources
An agent can read code, search issues, or inspect Git history within its current task without preserving the previous session. Cross-task memory adds the investigations already performed, failed approaches, the final fix, and its validation.
A commit may preserve only the final patch. A task record can also explain that moving a validation step was attempted and broke compatibility. A later agent working on the same behavior can use that record to investigate the relevant constraint.
In a memory comparison, give both groups the same current code and tools, then vary access to previous task experience. Restricting the baseline's normal repository access would mix a tool-access difference into the memory comparison.
Measure resolution and the cost of reaching it
| Paper's evaluation dimension | Question |
|---|---|
| Resolution accuracy and its change | Does prior context help solve related tasks or interfere with them? |
| Time efficiency | Does it reduce repeated investigation and unsuccessful attempts? |
| Token and cost efficiency | How many tokens does solving the related task consume, and how does cost change? |
Teams can also assess architecture consistency and explanation quality using separate criteria alongside the paper’s metrics.
When past experience helps, and when it misleads
The paper reports benefits from accurately summarized and retrieved experience, while unfiltered or incorrectly selected context can provide limited or negative benefits.
Applying this finding requires checking why a record fits a task. Similar error messages may have different causes. An old fix to the same function may depend on an interface that has since changed. Version, module, and validation details give the next agent evidence to inspect.
Apply the idea to a coding product
Start a product evaluation with a pair of related tasks. The first produces experience; the second uses it. Hold the second task's starting code, model, tools, and budget constant, with and without the earlier experience.
Memory for the second task must contain only information available before that task began. Its answer, patch, and later test feedback cannot be included in the memory used to solve it.
Retain both helpful and misleading cases. A helpful case should identify the record that avoided a particular investigation. A misleading case should expose the condition that was applied incorrectly. State whether timing includes the initial cost of producing experience, and distinguish one use from repeated reuse.
Decide whether the experience is worth keeping
Experience has measurable value when it improves completion, or reduces time and cost at similar completion. If the system retrieves more history without improving outcomes, inspect record selection, missing conditions, and whether the task was already straightforward from current code.
This evaluation approach can be applied to MemoraX Code and other coding memory systems. Comparisons should use the same tasks and report both outcomes and their cost.
References https://arxiv.org/html/2602.08316v3
