When retrieval should retry, abstain, or ask for review

A retrieval failure is a decision problem

A question enters a retrieval system and comes back with plausible passages. The tempting next step is to ask a language model to produce an answer. The more useful question is whether those passages are sufficient for the requested claim. Some failures need another search. Some need a narrower question. Some require a person who can resolve a conflict. Others should stop immediately because the evidence is unavailable or forbidden.

This essay extends the engineering questions in my existing Corrective RAG article. It specifies a study protocol rather than announcing a validated controller. The goal is to compare a simple, inspectable decision policy with increasingly learned alternatives while preserving the reasons for every branch. A more elaborate evaluator is only useful if its decisions improve the system under the costs that matter.

The original Corrective Retrieval Augmented Generation paper introduced a retrieval evaluator that triggers corrective actions, including supplementary web retrieval, and a decomposition/recomposition method for evidence. The retry, abstention, authorization, and human-review policy developed here is an application design inspired by that problem. It should not be presented as an exact reproduction of the paper or as an already measured improvement.

Separate relevance, sufficiency, and permission

A passage can be relevant without answering the question. A policy mentioning retention periods may omit the exception that applies to a particular record. A passage can answer the question while being inaccessible to the requesting user. A public document can be accessible but superseded. One confidence number cannot communicate these different situations without hiding important decisions.

Define the evaluator's outputs as structured observations: candidate relevance, coverage of required facts, agreement among authoritative sources, revision validity, and authorization status. Permission is a hard boundary handled by a trusted policy layer. It is not an evaluator score that can be outweighed by high relevance. The evaluator should only receive material the request is permitted to expose to it.

For an illustrative question about an equipment procedure, required facts might include the equipment revision, operating condition, and procedure version. The evidence can be topically excellent while lacking the revision match. A safe response explains that gap. It should not quietly reinterpret a specific question as a generic one. That is why the query contract belongs next to the retrieval contract: sufficiency depends on what was actually asked.

Make corrective actions bounded and distinguishable

A retry should have a reason and a changed plan. Repeating the same query against the same snapshot is not meaningful progress. A bounded controller can rewrite an ambiguous term, expand an approved source collection, fetch a missing parent section, or request a current revision. Each action consumes an explicit budget and records what new evidence it expects to obtain.

Human review is appropriate when a person can resolve something the system cannot, such as two approved documents with conflicting applicability. Abstention is appropriate when the available information cannot support the requested output and no useful authorized action remains. Clarification is appropriate when the question lacks a parameter that changes the answer. These outcomes should have different user messages and different operational queues.

The following pseudocode shows the branch structure. The evidence checks and permission decision are assumed to be established upstream; the example is not a complete security boundary. A production policy also needs cancellation, deadlines, and durable records so that a disconnected client does not leave corrective work running indefinitely.

1def next_action(state):
2    if not state.authorized:
3        return "deny"
4    if state.required_parameter_missing:
5        return "clarify"
6    if state.authoritative_conflict:
7        return "review"
8    if state.support_complete and state.citations_valid:
9        return "answer"
10    if state.retry_budget > 0 and state.new_search_plan:
11        return "retrieve_again"
12    return "abstain"

A worked example: the wrong revision looks convincing

Suppose an approved corpus contains two revisions of a procedure. A question asks about the current revision, but a dense retriever ranks a detailed paragraph from the older one above a terse replacement. Both paragraphs use the correct terminology. A relevance-only evaluator is likely to miss the important disagreement because relevance is not the failure.

The first controller step checks the source revision against the query contract. It rejects the older paragraph as sufficient evidence for the current answer while retaining the retrieval event for debugging. A corrective search then requests the current section and its applicability note. If both arrive and support the answer, generation can proceed with current citations. If the note remains unavailable, the system reports exactly which condition could not be established.

This case should appear in the evaluation set in several forms: a clearly obsolete document, an unlabeled revision, conflicting metadata, and an inaccessible replacement. They exercise different dependencies. A metadata repair might fix the first two; a better embedding alone may fix none. The experiment needs to reveal that distinction rather than credit every successful answer to the retrieval model.

Evaluate the controller at a declared operating point

Build a held-out set containing answerable requests, incomplete evidence, conflicts, stale sources, ambiguous requests, and policy-denied cases. Label what an acceptable action would be as well as what an acceptable answer would contain. Keep related document revisions and near-duplicate questions in the same partition so that a familiar example does not masquerade as independent evidence.

Start with fixed rules and a single retrieval pass as separate baselines. Add corrective retrieval, then add the learned evaluator, then citation verification. This ordering exposes whether gains come from another opportunity to search, better routing, or stricter answer filtering. Measure unsupported answers, useful answer coverage, inappropriate abstentions, review demand, latency, and cost together. An abstain-everywhere policy can appear excellent if the only reported metric is the error rate among answered questions.

Thresholds belong to a development set. If an evaluator emits probabilities, check their calibration before using expected-cost decisions. Scikit-learn's calibration guide explains reliability diagrams and the distinction between a classifier's scores and calibrated estimates. For this study, the key practical requirement is to freeze the chosen threshold before the final evaluation and report the operating point, including how many requests it declines.

Preserve the reason for every non-answer

Observability should reconstruct the decision without exposing more source content than necessary. Record the query contract version, corpus snapshot, retrieval configuration, authorized candidate identifiers, evaluator version, chosen action, and terminal reason. Store sensitive traces under appropriate access controls. A public dashboard does not need raw documents to explain that a request stopped because its required revision was missing.

The most useful error review compares neighboring cases: an unsupported answer beside an appropriate abstention, or an unnecessary review beside a successful correction. That comparison shows whether the controller understands a meaningful distinction or merely reacts to superficial patterns. Review overrides are valuable evidence, but they should not flow directly into training without checking their consistency and source of authority.

No controller removes the need to maintain the corpus. If every retry fails because the approved source is absent, the problem is ingestion or ownership. The research outcome may therefore be a simpler policy and a better document contract, rather than a larger evaluator. That is a useful result. The purpose of corrective retrieval is to improve decisions under imperfect evidence, including the decision to stop.