Local on-device AI.
Rivaling frontier models.

What this test supports.

Our training and app lift the model we started from, full-precision base Gemma E4B, by 10 points on meaning. Across all 21 languages and all lengths the judges differ: Sonnet finds no clear meaning difference from Haiku or 31B, while Gemini rates both clearly higher. Use the controls below to see each view.

Why 40 words matters.

A real health conversation moves one turn at a time: one person says a sentence or two, the translation plays, and the other person answers. Those turns usually run under 40 words, so the ≤40-word results reflect real use more closely than longer passages, and this page opens on them. OpenTalk is built for exactly this: most of its training examples are short health conversations, and its prompt, health-term hints and input rewrites are tuned to how people actually speak, not to long documents.

Compare the outcomes. Not a winner’s podium.

Meaning preservation is the primary endpoint on this page. Critical flags remain a separate risk signal. Readability is available as a diagnostic, with no fluency-driven overall ranking.

Two judges, read separately.

Both judges saw the same shuffled, letter-labeled packets and worked independently. They agreed on the meaning check in 93.6% of cases (Cohen’s κ 0.63) and on critical flags in 97.3% (κ 0.53). Gemini flagged 778 outputs as critical and Sonnet 340; both flagged the same 301. That gap applies to every system, so critical-flag rates should not be read as stable counts. Neither judge is a reference standard. “Both judges, averaged” averages the two on each translation, so a pass from one and a fail from the other counts as half. “Both judges, agreed” counts a finding only if both judges made it, on the same translation though not necessarily for the same error. It removes one judge’s false alarms and also real errors only one judge caught, so it is the most lenient view.

How sure we are.

Each line shows OpenTalk’s gap to another system, with both judges averaged, and the likely range of that gap at 95% confidence. These comparisons pair the same source text and locale, and the intervals resample whole source texts. Right of zero favors OpenTalk. A filled dot means the whole range sits on one side of zero, so the difference is clear. A line crossing zero shows no clear difference. It does not prove the systems are equal. They describe sampling uncertainty given the judges’ ratings, not judge bias.

Fluency is a separate finding.

Haiku has the highest recorded fluency-based pass rate under both judges. That is a result of their assessments, not a demonstrated comprehension advantage. The judges disagree most about 31B: 83.5% with Sonnet and 93.2% with Gemini at the 4/5 threshold. These thresholds materially change the figures, so we do not turn them into a general quality tier.

The prompt asks for a native-naturalness score from 1 to 5, ignoring accuracy, without behavioral anchors for the 3/4 boundary. Harmless bilingual drug names and stylistic preferences may affect ratings. 31B’s repeated drug names may explain part of how Sonnet rated it; Gemini rated it much higher. This is a review concern, not an adjudicated score correction.

What was measured.

• Material: 150 English texts from routine health conversations, 15–82 words, each translated into all 21 locales. There are 3,150 outputs per system. This tests written text translated from English, not complete two-way spoken conversations.
• Meaning check: no critical or major non-form error, no required key item wrong or missing, and usable as delivered. Minor or cosmetic issues may remain; passing is not a guarantee of perfect meaning or safety.
• Serious-error outputs: the percentage containing at least one major or critical non-form error. This counts affected translations, not individual errors.
• Potentially harmful outputs: outputs containing at least one non-form error labeled critical by the AI judge, expressed as a percentage of outputs. These are a subset of serious-error outputs. No real-world harm was measured.
• Key facts wrong: the percentage with at least one required checklist item wrong or missing. These can overlap with serious and critical flags; the measures are not added together.
• Fluency beneath the meaning bars: the mean naturalness rating divided by 5 and multiplied by 100. This is a percentage of the maximum rating, not the share of accurate or fluent outputs. It is not combined with meaning preservation or used to rank systems.
• Readability diagnostic: meaning check passed plus a fluency threshold. The default diagnostic uses 4/5; the alternatives are shown above.
• Judges: Claude Sonnet 5.5 at medium effort and Gemini 3.8 Flash at high effort, judging separately with the same rubric. Headline figures average the two on each translation. Candidates were shuffled and letter-labeled, without model identities. Judges were instructed not to infer authorship or penalize correct alternative wording. Blinding does not establish that style or model-family bias is absent.
• Core languages: Arabic, Brazilian Portuguese, European Spanish, French, German, Hindi, Italian, Latin American Spanish, Russian, Simplified Chinese, Traditional Chinese, Vietnamese. The page and its opening figures use these 12 by default. All 21 remain available, and the group was chosen after these results were known.
• Development exposure: Eval A (originally Batch 1) contains 90 texts. Some errors there informed app fixes and version selection. Eval B (originally Batch 2) contains 60 texts; no fixes were built from those texts. Neither evaluation set has human adjudication.
• Generation procedures: systems were scored as delivered. Updated OpenTalk cases were rejudged by Sonnet after app fixes; other systems did not receive that opportunity. Gemini judged the final set once. One overwritten Haiku output was counted as delivered and unusable under both judges.
• Weighting: each text/locale case counts equally. This is a benchmark mix, not actual app-traffic weighting. The 21-locale stress test should not be mistaken for a guarantee across every supported language.

An exploratory combined analysis.

The limits belong beside the results.

• Two AI judges, no human reference: Sonnet and Haiku share a provider, Anthropic. Gemini and the Gemma models, including OpenTalk’s base model, share a provider, Google. Claude also helped run the evaluation and draft these pages. Blinded bilingual human review is needed before treating these labels as validated.
• No retrospective score corrections: disputed Persian judgments are still included as originally scored. Any adjudication should follow the same documented rule for every system.
• Exploratory comparisons: many subsets were examined, including the core-language group. Intervals are not adjusted for every comparison or for version/endpoint selection. Single-locale rows are small; they are descriptive and have no paired intervals here.
• Longer and difficult texts: over 40 words, OpenTalk passed the meaning check in 92.8% with Sonnet and 89.9% with Gemini, compared with 95.0% and 93.8% for Haiku and 96.5% and 96.5% for 31B. On the ten difficult source texts, OpenTalk passed in 87.1% and 86.7%. These subgroup rates need their uncertainty and context, not a blanket winner label.
• Outside this test: real-world comprehension, translation back into English, speech recognition, dense long passages, and iPhone latency, battery and reliability were not measured. Local model size is system context, not a quality score.
• Use: OpenTalk is a translation aid, not a replacement for a live interpreter service. Check important details, especially names, doses, numbers and instructions.

Inspect the underlying records.

Original v13 Sonnet-run exports, with original scores unchanged, and Gemini’s scores and flags as recorded. The full report includes an older custom composite; it is not the primary score on this page. The original report’s broader “clean” cut is different from this page’s Eval B.