Understand ScriptMem tasks, a concrete question, and the published MemoraX results, including the difference between score verification and full reproduction.
ScriptMem tasks cover characters, events, temporal relationships, and reasoning across passages. ScriptMem is a long-term memory benchmark published in the official memorax-ai repository. It builds tasks around characters, events, and relationships in scripts, testing whether a system can assemble the evidence needed to answer a question. Version 1.0 contains 457 questions.
What scripted narratives contribute
Characters recur across scenes, discuss the same events, and add or revise information. This supports tasks that distinguish people, places, times, and relationships. These narrative tasks complement conversational evaluation. Effects on code changes and real user preferences need evaluation in those settings.
ScriptMem groups questions into User Profile, Event Tracking, Temporal Evolution, Social Relations, Fine-Grained Data, and Lessons Learned. Results by category show which tasks a system handles well and where it makes mistakes.
| Answer format | Questions | Scoring requirement |
|---|---|---|
| Single choice | 298 | The correct option |
| Multiple selection | 123 | The exact set, with no missing or extra options |
| Ordering | 36 | The exact sequence |
The categories describe what a question tests; the answer formats determine how it is scored. Accuracy on selection and ordering tasks does not directly measure open-ended question answering.
A question that combines several clues
Public question angry:conv-0#q0000 asks what a demonstration establishes, given a door-opening action, an apartment layout, and testimony. It requires distinguishing an internal door from the entrance to an external hallway, then relating the chain's location to the action. The full question is in the public data.
Retrieving only that somebody opened a door would leave the question unresolved. To diagnose a wrong answer, follow the retrieved evidence and reasoning: did the system identify the wrong door, miss a spatial relationship, or misinterpret the clues it found?
Reading the reported 60.3% result
The project's May 2026 leaderboard reports the following results under a shared GPT-4o-mini backbone. MemoraX here identifies the memory system evaluated in that table.
| System | Overall accuracy |
|---|---|
| MemoraX | 60.30% |
| EverMemOS | 42.90% |
| Mem0 | 42.00% |
| MemOS | 36.40% |
| M-Flow | 32.60% |
The lead over EverMemOS is 17.4 percentage points, or approximately 40.6% relative to its score. Relative improvement and percentage-point difference are different ways to express the same comparison.
Overall accuracy describes performance under this evaluation setup. Identifying whether the advantage comes from temporal reasoning, character distinctions, or retrieval requires category results and execution traces. Coding performance for MemoraX Code requires a separate coding evaluation.
How to check the evaluation results
First check the system versions, model configuration, and budgets for writing, retrieval, and answering. Per-question predictions can then be passed through the same scorer. Exact matching matters particularly for multiple-selection and ordering questions.
The repository releases questions, reference answers, submission formats, and scoring scripts. It does not distribute the original script conversations. Rescoring existing predictions and rerunning a complete memory system are therefore separate procedures. Full reproduction also needs the corresponding source texts and preprocessing.
A worked error showing the source evidence, retrieved material, and selected answer can identify an actionable change: preserve an omitted fact, repair an object association, or adjust retrieval and answer generation.
References https://github.com/memorax-ai/ScriptMem/blob/main/README.md https://github.com/memorax-ai/ScriptMem/blob/main/data/public/questions.jsonl https://github.com/memorax-ai/ScriptMem/blob/main/data/README.md
