Conversation design · Explore this field ↗ · Implementation · 4 min read
Long-conversation context budgets: allocate, do not just trim
A long conversation needs explicit space for current intent, pinned constraints, verified evidence, and a fallible summary—not a mystery pile of old turns.
Turn context assembly
- 01Read latest turn
- 02Load pinned state
- 03Retrieve allowed evidence
- 04Fit budgeted summary
- 05Record manifest
A summary is a fallible derived lane, not ground truth.
Original conceptual diagram · not a live trace or measured result.Budget by purpose
Do not describe context as one bucket. Reserve separate lanes for immutable system and safety rules, current task state, the latest user message, confirmed user preferences, retrieved evidence, and a compressed history. State a maximum for each lane and an overflow rule. Recent language is not automatically the most important material; a confirmed delivery restriction may matter more than six friendly exchanges.
The first design question is what must survive a turn boundary. “Everything” is not an answer because it creates privacy, cost, and attention problems. A budget forces the team to name which facts are durable and which merely help conversational tone.
Pin constraints, not guesses
A pinned item should have a source, scope, owner, timestamp, and invalidation rule. Good candidates include a user-confirmed language preference, a current authorization scope, a policy version, or an explicit constraint such as “do not cancel the appointment.” Do not pin model inferences such as a guessed product, mood, or diagnosis. Label those as hypotheses and ask again when they become consequential.
Hypothetical worked example: a user initially asks to reschedule for Tuesday, then corrects it to Thursday. The current task state records Thursday as confirmed and marks Tuesday superseded. The history summary may mention that a change occurred, but it must not continue presenting Tuesday as an active slot.
Summaries are derived data
Generate a summary as a versioned artifact, not invisible prose. Include source-turn range, generated time, facts retained, unresolved questions, and statements explicitly marked uncertain. Give a reviewer a way to compare the summary with the underlying authorized turns. If a user corrects a fact, update or invalidate the summary immediately instead of appending another contradictory sentence.
A summary can also be wrong through omission. Keep a small test set where an exception, refusal, handoff promise, or user correction occurs before compaction. The test asks whether the later conversation preserves the decision. It does not demand identical wording from a language system.
Assemble evidence late and narrowly
Retrieval evidence should answer the current question, not become permanent conversation memory. Apply authorization and freshness filters, record its source version, and remove it from the next turn unless the task needs it. A long document pasted into every turn can crowd out the user’s latest request while increasing the chance that stale clauses look authoritative.
Use a context manifest for debugging: lane, item ID, token estimate, inclusion reason, policy state, and omission reason. Do not log full private content solely to count tokens. Operators can then see whether a bad answer came from a missing constraint, an overlong evidence block, or an inaccurate summary.
Make privacy and retention concrete
Context selection is a privacy decision because a durable summary often outlives the displayed chat. Set retention by purpose, let users correct or remove stored preferences where the service offers that control, and avoid carrying sensitive details into unrelated tasks. A system that remembers a health disclosure to make a shopping recommendation feel personal may be technically capable and still inappropriate.
NIST AI RMF frames trustworthy AI work around context-dependent risks and governance. Use that as a reminder that an elegant token budget is not enough: the allowed purpose, data handling, and failure consequences must be reviewed by the service owner.
Release with context fixtures
Test a full budget, a correction just before compaction, conflicting historic and current facts, a revoked document, a handoff return, and a new task in the same session. Assert that essential pinned constraints remain, stale values are omitted, citations reflect current evidence, and an omitted item has a traceable reason. Set success criteria in behavior: no action uses a superseded confirmation; no shared context crosses task or tenant scope; and reviewers can reconstruct the input lanes.
The limitation is simple: compaction loses detail. A budget does not solve that; it makes loss intentional, inspectable, and recoverable through a source lookup or human handoff. Record the fixture outcome with the context manifest, so later prompt or model changes can be compared against a named allocation rather than a vague transcript. Keep the manifest with the approved test case.
Take it into the review
Context manifest
| Lane | Admission rule | Expiry | Example |
|---|---|---|---|
| current task | confirmed state only | task close | Thursday slot |
| pinned constraint | explicit source | policy change | authorized account |
| evidence | current permitted source | next decision | returns clause |
| summary | reviewable derived record | new correction | unresolved question |
A starting artifact to adapt to your service—not a ready-made policy, compliance certificate or test result.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.
What the sources establish
Artificial Intelligence Risk Management Framework 1.0
NIST AI RMF describes risk management and trustworthy AI characteristics as context-dependent concerns that require governance and measurement.
Limits: It does not prescribe a language-model context window or summary algorithm.
Checked 2026-09-19 · NIST · source publication date not established.
Open original source ↗OWASP Top 10 for Large Language Model Applications
OWASP identifies risks in LLM application design that need controls beyond model prompting.
Limits: It does not define a universal context allocation pattern.
Checked 2026-09-19 · OWASP · source publication date not established.
Open original source ↗Procedures and worked examples are editorial synthesis. Preparation/review dates are not claimed historical publication dates.