Corpus chunking evaluation: test boundaries before tuning retrieval

A model to inspect

Compare evidence boundaries

01Inspect source layout
02Build chunk variants
03Retrieve fixed fixtures
04Check required evidence
05Review source locators

Compare variants against the same source versions. A fluent answer cannot compensate for missing evidence.

Original conceptual diagram · not a live trace or measured result.

Inventory structure before splitting

Chunking starts with the source layout, not a token number copied from another corpus. Inventory headings, paragraphs, tables, captions, footnotes, repeated headers, lists, page breaks, and structured fields. Preserve a parent document ID, source locator, heading path, and extraction method for every candidate chunk. A retrieval result without a recoverable parent and location cannot be reviewed when an answer seems incomplete.

Microsoft’s document-layout guidance describes chunking based on document structure, including headings and semantically coherent paragraphs. It supports the narrow point that layout can be material to chunk boundaries. It does not establish that one layout model, overlap, or chunk length will work for a different corpus, language, or retrieval stack.

Make boundary choices testable

Define candidate policies before judging them: heading-first with a maximum size, paragraph packing with carry-forward heading, fixed windows with overlap, table-as-unit, and separate treatment for captions or definitions. Record exactly how each policy handles a section that exceeds its maximum. Overlap is not automatically good; it can preserve a sentence boundary while duplicating evidence, inflating index size, and making a repeated clause appear more authoritative than it is.

Create a controlled dataset with known answer evidence. Include a definition split across a heading and its first paragraph, a table whose condition and exception are in different cells, a list with a final caveat, a page header repeated in extraction, and two nearly identical documents with one changed threshold. These are retrieval fixtures, not claims about real user prevalence.

Judge evidence coverage, not prose

For each fixture, record the minimum source material necessary to answer correctly and the source locators that contain it. A chunking variant passes a retrieval case only when the retrieved evidence covers those required units; a fluent answer produced from half the rule does not pass. Separate retrieval recall, evidence coverage, citation locatability, and answer behavior so an apparent answer improvement cannot hide a missing exception.

Hypothetical worked example: a benefits table says an allowance applies only after a qualifying period, while a heading defines the qualifying group. A chunk that returns the amount but not the group is incomplete evidence. The evaluation should flag it even if a model guesses the missing condition correctly.

Inspect tables and extraction failures

Tables deserve their own fixtures because reading order, merged cells, row headers, footnotes, and units can be lost before chunking begins. Preserve a table identifier, column labels, row labels, and nearby notes when possible. If the parser cannot establish the relationship between a number and its qualifier, do not flatten it into a prose chunk that implies certainty. Route it for extraction repair or exclude it from answerable evidence.

Counterexample: overlapping text windows duplicate a footnote in several chunks. A ranker may repeatedly retrieve the footnote while dropping the row it qualifies, creating a persuasive but unusable answer. Diagnose the extraction and boundary record before tuning the retriever; more overlap cannot reconstruct a missing table relationship.

Compare variants under release discipline

Hold out some fixtures while designing a chunker, then compare candidates using the same queries, filters, and source versions. Record retrieval count, required-evidence coverage, false positive sources, and the exact chunk IDs inspected. Review failures by document shape, not only aggregate score: a policy manual, spreadsheet export, and narrative article can need different handling.

Acceptance checks: every chunk retains a parent and locator; headings and table semantics have explicit rules; the controlled set includes boundary and extraction cases; and a selected variant has documented wins and regressions. Chunking can improve retrieval representation, but it cannot supply absent documents, fix unauthorized access, or prove answer correctness.

Keep the source view available

Every evaluation review should let a reader open the original source region alongside the extracted chunk and its parent metadata. This is how a team detects a parser that changed table order, lost a footnote, or merged headings from adjacent documents. Store a source fingerprint so an old fixture is not accidentally evaluated against a revised file. When the fingerprint changes, treat cached extraction results as requiring revalidation before they are reused in a comparison.

RFC 9111 supplies the narrow cache vocabulary: a changed validator makes a stored response stale for reuse until validation. It does not define corpus-version policy, but it supports making source-version revalidation an explicit test condition. A passing retrieval result is only as inspectable as its path back to the source. When the source cannot be displayed or its location cannot be trusted, classify the fixture as extraction-limited rather than claiming evidence coverage.

Take it into the review

Chunking evidence matrix

Fixture shape	Boundary policy	Required evidence	Retrieved chunk IDs	Coverage decision
heading + definition	carry heading into paragraph	term and definition	c-17,c-18	pass only if both
table exception	table as unit	row, column, footnote	c-42	manual review
repeated header	remove extractor header	body clause only	c-63	exclude header
near-duplicate policy	parent/version retained	current threshold	c-81	reject stale parent

A starting artifact to adapt to your service—not a ready-made policy, compliance certificate or test result.

Primary reading

Sources and limits

These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.

What the sources establish

Azure AI Search: Chunk by document layout

Azure AI Search documents structure-aware chunking using headings and semantically coherent paragraphs and sentences, with parent and chunk fields.

Limits: It is product documentation, not evidence for a universal chunking configuration or evaluation result.

Checked 2026-09-19 · Microsoft · source publication date not established.

Open original source ↗

RFC 9111: HTTP Caching

RFC 9111 defines HTTP cache invalidation and validation behavior.

Limits: It does not define conversational authorization, source lifecycle, or model memory.

Checked 2026-09-19 · IETF · source publication date not established.

Open original source ↗

Procedures and worked examples are editorial synthesis. Preparation/review dates are not claimed historical publication dates.