Multilingual evaluation fixtures: test the interface, not only translated sentences

Define fixtures as interaction cases

A multilingual fixture is more than a prompt and an expected response. Record the locale, script, text direction, input method, device width, relevant date or number format, task state, expected system action, and expected human-review outcome. Include a reason the case exists: ambiguous pronoun, borrowed product term, line wrap, right-to-left plus left-to-right identifier, or a policy phrase that must remain exact.

This extends a general warning that multilingual is not translation. The objective is not to rank languages against English. It is to catch a visible or operational mismatch before a user has to explain it. Keep source and localized fixtures connected, but permit a locale-specific case when the failure cannot occur in the source language.

Cover direction and mixed content

Test dir=rtl interfaces with Arabic or Hebrew text, long labels, punctuation around URLs, numbers, product codes, dates, icon placement, focus order, and error messages. Do not assume that flipping a layout fixes bidirectional text. A case with an embedded English identifier in an RTL sentence can expose different behavior from an all-RTL response. Make expected visual order explicit in the fixture screenshot or review note.

Worked example, hypothetical: an Arabic fixture asks to amend reference AB-204 on a date. The expected result preserves the reference as an identifier, renders the action label and error in Arabic, and keeps the keyboard focus sequence meaningful. The test does not declare that Arabic speakers prefer one universal wording; a named reviewer marks wording uncertainty for follow-up.

Test local data as well as language

Include dates with ambiguous numeric order, local numerals where supported, decimal and grouping conventions, names with multiple parts, address formats, currency labels, time-zone display, and plural-sensitive quantities. Verify that retrieved knowledge has the correct locale or an honest fallback, rather than translating a source it does not support. Never use a localized reply to hide that a policy is available only in another language.

The counterexample is a fixture suite that has elegant translated greetings but no test for a deadline, a telephone number, or a right-to-left handoff link. Such a suite measures fluency while missing task completion.

Run a review process, not a one-time score

Assign reviewers who can assess the relevant language and service context, but do not invent credentials or treat one person as a whole locale. Provide a checklist: meaning, harmful ambiguity, terminology, direction, rendering, keyboard use, source availability, and handoff route. Reviewers may mark “needs policy decision” or “cannot verify” rather than forcing a pass. Maintain a decision log when source language and local adaptation conflict.

Acceptance checks: fixtures cover each supported locale and critical task; every fixture declares direction and data conditions where relevant; a change reruns affected fixtures; and failures link to an owner and retest. Report coverage as fixture coverage, not proof of cultural adequacy or model fairness.

Release check. Require a reviewer decision or explicit unresolved flag for critical task fixtures. Missing review is a release condition, not an invitation to borrow an English result.

Keep limitations visible

Language tags and Unicode locale data help software choose presentation rules, but they do not capture every regional convention, dialect, accessibility need, or domain term. Automated comparisons can catch missing strings and broken variables, not whether a conversation is respectful or safe. Build a route to human support when the requested language or evidence cannot be handled.

W3C’s internationalization work and Unicode CLDR are technical references for direction and locale data. They are not an endorsement of specific translations. The fixture taxonomy and review method here are original synthesis and should change with observed failures.

A localized fixture should separately inspect retrieved citations, generated connective text, UI labels, and action parameters. A response may have fluent prose while its cited source is in another language, its button labels overflow, or its tool request carries a differently formatted date. Record which layer failed so a translation team is not asked to fix a retrieval policy and engineers are not asked to rewrite reviewed terminology. This layered evidence makes reruns more useful after a source or interface change.

Include a change-control fixture for a fallback: the system should say when no reviewed local source exists, keep the original source identifiable, and offer an approved alternate route. A fluent unverified paraphrase is not an acceptable substitute for evidence in a critical language path.

Version the evidence behind each case

Each fixture should point to the policy, knowledge source, or intended product behavior it checks. When that evidence changes, flag localized variants for review instead of trusting an old expected answer.

Use production-like but non-personal examples. This protects reviewers and prevents fixture data from becoming a hidden copy of real conversations. Repeated failure around one term is useful improvement evidence, not proof about an entire language.