Retrieval abstention test set: make no-evidence measurable

A model to inspect

Make the evidence gap testable

01Declare permitted sources
02Withhold or contradict evidence
03Run the fixture
04Inspect claims and citations

The expected result may be clarification, a scoped gap or a handoff—not an invented answer.

Original conceptual diagram · not a live trace or measured result.

Make unanswerability a fixture class

A grounded system needs questions it should not answer from the available corpus. Label fixtures as answerable, source-missing, contradictory, stale, unauthorized, or out of scope. Each fixture names the permitted sources, the expected visible behavior, and a prohibited assertion. The release gate should reward a bounded explanation of the gap, a clarification, or a handoff when appropriate—not a polished completion without evidence.

The OpenAI Evals guide supports structured evaluation workflow, not an abstention policy. Rajpurkar, Jia, and Liang’s 2018 SQuAD work supplies a useful narrow precedent: plausible unanswerable questions can test whether an extractive system abstains when a paragraph lacks support. It is not a modern RAG guarantee. Your fixture’s source scope, authority rule, and release threshold remain product decisions that must be visible.

Separate no source from conflicting source

A missing source asks the system to say what it cannot establish. A conflict asks it to identify the discrepancy and avoid selecting a winner without a declared authority rule. Keep them separate: merging both into a generic “I do not know” hides whether ingestion failed, the corpus is inconsistent, or the user lacks access. Record the expected citation behavior for each class.

Hypothetical example: one current policy says standard returns have a stated window; no source covers custom items. The custom-item fixture expects a scoped statement that the exception is not established. A separate contradiction fixture supplies two active documents with different windows and expects escalation or an authority lookup, not an average or a confident choice.

Control what the grader can observe

A release fixture should evaluate both the response and the evidence path. Capture the query, allowed source IDs, retrieval result IDs, expected action class, prohibited claim categories, and a reviewer rationale. A simple deterministic check can assert that no excluded citation appears; a human or carefully specified grader can assess whether the response overstates support. Do not grade exact wording when several honest explanations are acceptable.

Avoid leaking the desired answer into the prompt through hidden fixture text. Keep expected evidence separate from runtime context and run the same fixture after changes to chunking, ranking, prompt policy, source ingestion, and authorization filters. A test that exposes the answer in its setup only measures copying.

Include hard negative controls

Negative controls should resemble answerable questions closely enough to expose unjustified completion. Use a known document title with an absent clause, a superseded source excluded by lifecycle state, a plausible but nonexistent product feature, a question that crosses tenant scope, and a request whose required source is intentionally withheld. Do not invent real people, cases, or incidents to make the fixtures vivid; use roles and synthetic identifiers.

Counterexample: a test set contains only obvious nonsense questions. A system can refuse those while still fabricating a missing policy exception phrased like a normal support question. The better control is semantically plausible and has an explicit evidence boundary.

Release on visible uncertainty

Review pass rates by fixture class and inspect every high-impact failure, especially a confident answer where the fixture required abstention. Acceptance checks: each negative fixture has a documented reason, current source scope, expected action, and prohibited assertion; evaluation logs identify the corpus version; and a change cannot replace an abstention with unsupported prose without review.

The limitation is that a finite test set cannot enumerate all ways evidence can be missing or contradictory. It is a repeatable release guard, not certification of truthfulness. Maintain it with source owners, add failures that teach a distinct boundary, and retire fixtures only when their documented premise no longer exists.

Protect the release signal

Keep the abstention set outside prompt examples and product demonstrations that could teach the system its expected reply. Review changes with a diff that shows the fixture premise, allowed sources, and grader rule. A new negative fixture should describe a distinct boundary, not duplicate an old sentence with a new identifier. Preserve the reviewer’s reason for the expected abstention so later editors cannot silently convert a source gap into a preference.

Use failures to improve either corpus scope, retrieval behavior, response policy, or evaluation design, and record which changed. Otherwise a pass may reflect a weakened test rather than a safer system.

Take it into the review

Abstention fixture register

Fixture class	Allowed evidence	Expected action	Prohibited claim	Review signal
source missing	none	state gap	invented exception	no citation
contradictory active rules	two conflicting IDs	escalate or authority lookup	choose unsupported winner	conflict visible
unauthorized material	excluded source	neutral access boundary	title or excerpt leak	no excluded ID
superseded source	current source absent	state unavailable	use retired rule	lifecycle checked

A starting artifact to adapt to your service—not a ready-made policy, compliance certificate or test result.

Primary reading

Sources and limits

These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.

What the sources establish

OpenAI guide: Working with evals

The Working with evals guide describes evaluation workflow and structured evaluation configuration.

Limits: It does not prescribe abstention behavior, a dataset, or a passing threshold.

Checked 2026-09-19 · OpenAI · source publication date not established.

Open original source ↗

Know What You Don’t Know: Unanswerable Questions for SQuAD

Rajpurkar, Jia, and Liang present an extractive reading-comprehension benchmark with adversarially written unanswerable questions that require systems to abstain when no answer is supported by the paragraph.

Limits: It is a 2018 extractive benchmark, not a modern RAG evaluation or a guarantee that an abstention policy is correct.

Checked 2026-09-19 · Association for Computational Linguistics · source publication date not established.

Open original source ↗

Procedures and worked examples are editorial synthesis. Preparation/review dates are not claimed historical publication dates.