Evaluation & operations · Explore this field ↗ · Practice · 4 min read
Synthetic conversation test data: useful fixtures without invented users
Create clearly fictional, nonidentifying fixtures while measuring what their construction fails to represent. This field note supplies an inspectable artifact, counterexample, and release checks.
Part of the 20-guide fieldwork edition.
Keep a fixture’s fictional origin attached
- 01Define scenario coverage
- 02Write synthetic turns
- 03Check contamination
- 04Version expected behavior
- 05Label exported results
A passing fictional case tests a declared behavior; it does not measure real customer outcomes.
Original conceptual diagram · not a live trace or measured result.State the fictional origin plainly
Synthetic conversation fixtures are invented test material, not observed user behavior. Label every item with its synthetic origin, authoring method, date, intended scenario, and any source patterns it was designed to exercise. Use roles such as customer, support specialist, or room participant rather than realistic personal names, addresses, account numbers, or copied transcripts. A plausible voice is useful for exercising state transitions; it is not evidence about a population.
Start from a service contract: required fields, correction paths, failure states, action confirmation, and handoff boundaries. Then write turns that deliberately exercise those conditions. This keeps fixtures tied to a reviewable behavior instead of producing a large collection of charming but untestable dialogue.
Design a scenario distribution
List the dimensions the fixture set should cover: intent family, channel, language variant where supported, message length, correction, interruption, tool availability, safety boundary, and terminal state. Select proportions as an internal coverage plan, not a claim that they mirror production. Keep rare but high-consequence conditions visible even if they would be uncommon in ordinary traffic.
A hypothetical support fixture can ask for shipment status, correct an order reference, receive a retrieval miss, and choose handoff. Its value is that it checks identity binding and recovery in one controlled path. It must be tagged as hypothetical and must not be reported as a real customer journey, completion rate, or evidence of demand.
Prevent contamination and resemblance
Do not seed a synthetic dataset with private transcripts unless there is a separate, authorized data-governance process. Search generated fixtures for copied identifiers, distinctive phrases, account patterns, and source fragments before release. Keep synthetic test data separate from training, product analytics, and prompt examples unless a documented policy permits a specific transfer. A fixture can contaminate an evaluation if it appears in the prompt template, tuning set, or model-change comparison baseline.
NIST notes that synthetic data lacking differential privacy can have only informal privacy protection. That narrow warning means “synthetic” is not a privacy conclusion. For sensitive domains, have the responsible privacy and security reviewers define whether any source-derived generation is allowed and what testing is required.
Measure representativeness as a limitation
Synthetic data may miss vocabulary, accessibility needs, multilingual forms, error recovery, emotional stakes, and the uneven distribution of actual requests. Do not use its pass rate to estimate live satisfaction or demographic performance. Instead, record what it deliberately covers and what it cannot represent. When privacy-safe, separately compare the fixture taxonomy with aggregate, approved operational signals—not raw identity-rich conversation logs.
Counterexample: a dataset made only of complete grammatical turns makes a turn-taking system look robust while it never sees repairs, background noise, abandoned prompts, or overlapping input. Add those as explicit fictional cases. The absence of a real name does not automatically make a fixture representative.
Version, review, and retire fixtures
Version each scenario with its intent, setup state, expected observable events, allowed wording range, and reviewer decision. When a product rule changes, update the fixture’s declared premise rather than silently changing the expected output. Keep a changelog that distinguishes a new capability, a corrected test, and a removed obsolete scenario. This allows a regression review to ask whether the system changed or the measure moved.
Acceptance checks: every fixture is labeled synthetic; no item claims a real person or user outcome; coverage dimensions and exclusions are documented; and contamination review occurs before evaluation use. Synthetic fixtures are excellent for repeatability, but they complement rather than replace consented research, accessibility review, and monitored operational evidence.
Make synthetic labels survive export
Carry the synthetic flag and scenario provenance into every fixture export, dashboard, and result table. A file name alone is too easy to lose when datasets are combined. If an item is transformed, retain the parent scenario ID and the transformation reason so reviewers know it remained fictional. Keep a separate approval field for any source-derived material so a synthetic label never conceals a data-use decision.
This record also prevents a common reporting error: treating a synthetic pass as an observed operational metric. The safe label is “fixture result,” with the known scenario scope and its representativeness limits attached.
Take it into the review
Synthetic fixture card
| Scenario | Synthetic origin | Behavior exercised | Contamination check | Representativeness limit |
|---|---|---|---|---|
| order correction | written hypothetical role turns | state replacement | no real identifiers | not observed language |
| voice interruption | scripted audio-event labels | barge-in cancellation | not used in prompt seed | no acoustic realism claim |
| retrieval gap | fictional policy question | abstention behavior | no copied policy clause | not corpus coverage |
| shared handoff | invented support-role exchange | ownership transfer | separate from training | not satisfaction evidence |
A starting artifact to adapt to your service—not a ready-made policy, compliance certificate or test result.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.
What the sources establish
NIST SP 800-226
NIST discusses privacy and utility limits of synthetic data, including limits of non-differentially-private approaches.
Limits: It does not certify a conversational fixture as private or representative.
Checked 2026-09-19 · National Institute of Standards and Technology · source publication date not established.
Open original source ↗OpenAI guide: Working with evals
The Working with evals guide describes evaluation workflow and structured evaluation configuration.
Limits: It does not prescribe abstention behavior, a dataset, or a passing threshold.
Checked 2026-09-19 · OpenAI · source publication date not established.
Open original source ↗Procedures and worked examples are editorial synthesis. Preparation/review dates are not claimed historical publication dates.