Separate retrieval depth from search rounds. Learn when a larger top-k helps, when follow-up queries are needed, and how to compare quality and cost.

Retrieval choices: how many records to return, how to split the query, and whether to search again. Fixed top-k can also be used within a multi-round search.

Fixed top-k retrieval is a useful starting point for agent memory: return the k most relevant records, then pass them to the model. The difficulty is that tasks need different evidence. Finding a project convention may require one record; investigating a design change may require the original decision, its revision, and the current implementation.

A retrieval policy has separate choices: how many records to return, what to query, and whether to search again. Fixed top-k can be used inside an iterative search loop.

Result count and search control are separate decisions

A system that returns five records per query can stop after one query or revise the query and retrieve five more. The second workflow still uses a fixed k. It also follows up on missing evidence.

Increasing k from five to twenty expands the results for the existing query. It may not recover a record written in different terms, or one that becomes discoverable only after an earlier search reveals a filename.

Failure First adjustment What to inspect
Necessary evidence is just below the cutoff Increase k or improve reranking Useful additions versus repeated material
The query and record use different terms Rewrite the query or combine lexical and semantic retrieval Whether the record enters the candidate set
One clue is needed to find the next Add a retrieval round Whether the follow-up fills a named evidence gap
An old rule appears without its revision Check time, project, and version Evidence for the rule that currently applies

A project question that needs a follow-up

Suppose a developer asks: “Why did the payment module keep the old interface, and can we remove it now?” An initial search finds a migration plan saying that an external partner still depended on it. That explains the original decision. It does not establish whether the partner has since migrated.

The next query should investigate migration status, followed by a check of current callers. Twenty records about the old interface would still be insufficient if none establishes that migration is complete. The follow-up depends on a clue from the first query. Increasing k alone cannot replace that step.

A question about the repository's unit-test command may need only one current configuration record and the relevant repository file.

Iter-RetGen studies alternating retrieval and generation, using generated content to inform subsequent retrieval. It provides a research basis for iterative search, without establishing a universal number of rounds for long-term memory tasks.

Each round adds retrieval and model overhead. It can also follow a mistaken assumption. Record a specific gap, such as “partner migration status,” before searching again. Stop when the evidence supports the required conclusion. If the budget runs out first, identify what remains unresolved.

Compare policies on the same tasks

Hold the memory corpus, model, index, and scoring rules constant. Compare a small k, a larger k, and a policy that allows targeted follow-up queries. Include direct lookups, questions spanning multiple records, and conflicts between old and current rules.

Compare quality under similar total token or cost budgets, and compare cost at similar accuracy. Measure answer correctness, required evidence coverage, total input and output tokens, and end-to-end latency. Count intermediate model decisions in the iterative policy's cost.

If a larger k fixes the main omissions, keep that simpler design. If failures cluster around questions that depend on newly discovered clues, add an iterative loop for those tasks. Count how often each type of omission occurs before deciding whether to add retrieval rounds.

References https://aclanthology.org/2023.findings-emnlp.620/