Knowledge & actions · Explore this field ↗ · Operations · 3 min read
Retrieval debugging: inspect the source before tuning the answer
A source-first investigation loop for finding whether a bad grounded answer came from ingestion, chunking, retrieval, context assembly, or response generation.
See the source / a related case
Freeze one failing question
Begin with an exact question, the user’s permitted scope, the time of the answer, and the expected source or decision. Do not start by asking a model to improve the wording. Record the answer shown, its citations, request identifiers, retrieval query after any rewriting, collection or tenant filter, and configuration version. Redact personal data from the investigation record rather than removing the details that explain retrieval behavior. One stable case lets the team compare runs after a change; a loose complaint such as ‘the bot is bad at refunds’ usually mixes source freshness, missing metadata, ambiguous language, and response style.
Inspect the original before its chunks
Open the canonical source that should support the answer. Confirm title, owner, effective version, access class, formatting, and the exact passage—not a search snippet or an old export. Then inspect the ingestion record: extraction output, document identifier, page or section marker, chunk boundaries, metadata, embedding or index job, and completion state. Flowise documents that its Document Store retrieval can preserve original metadata and reconstruct documents from chunks; its custom retriever shows why the source field must be deliberately carried into context. Your own stack may differ, but the investigation still needs a durable link from the returned text back to an original source.
Read the candidate set, not just rank one
Run the exact retrieval query with the same authorization filter and save the top candidates, scores or order where available, document versions, and metadata filters. Look for three common patterns: the expected source is absent, it is present but cut at a harmful boundary, or it is present and loses to a less authoritative match. A source-inspection screen should show enough surrounding text to identify a dangling exception or table heading. Never use a score alone as an authority signal. Relevance ranking can be correct for wording while wrong for policy ownership, locale, lifecycle state, or effective date.
Separate retrieval from context assembly
A correct candidate can still vanish before generation. Compare the candidate set with the context actually sent to the model: token trimming, deduplication, reranking, metadata formatting, citation template, and system rules can change what survives. Flowise’s Agentflow retriever can return text with metadata, while plain text omits that inspection aid. Dify’s knowledge-retrieval workflow documentation describes a retrieval node whose output feeds later steps. In either product, treat source visibility as an observable contract: the response should not imply a citation exists if the relevant passage and identity were not actually supplied to the answering step.
Fix the narrowest responsible layer
If extraction dropped a table, repair extraction and re-ingest. If chunks separate a rule from its exception, adjust the segmentation and test nearby questions. If an expired document ranks, fix lifecycle filters or source governance before touching the prompt. If retrieved evidence is sound but the answer contradicts it, tighten the answer policy and add an evaluation case. Keep before-and-after traces, including a question that must remain unchanged, so a local repair does not create a broad regression. Do not manufacture a benchmark score: this workflow establishes diagnosis, not an unproven performance claim.
Make evidence usable to people
Operators need a compact evidence card: canonical title and URL, source version or effective date if known, chunk identifier, retrieved excerpt, query, filter outcome, and response citation. Users need a simpler version that communicates basis and limits without internal scores or hidden instructions. Escalate when the source itself conflicts, has no accountable owner, or is not authorized for the user. This editorial recommendation is to keep source inspection adjacent to the response lifecycle, not as a one-time launch exercise. Platform documentation establishes available inspection mechanisms; it does not certify that a particular corpus is complete, current, or correct.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.