# OpenTalk v13: full Sonnet results, 21 languages

**The test:** 150 clinic texts written for these rounds, 21 languages, 3,150 translations per system, 18,900 in total. One blind judge: Claude Sonnet 5.5 at medium effort. No human review. All results valid.

**Systems:** OpenTalk is v13 at 6/6/4 bits (5 GB) on the app's final route. Haiku is Claude Haiku 5.5 at low effort, asked plainly. 31B is Gemma 4 31B at full precision with the app's system prompt and no hints or checks. Qwen is Qwen 3.8 27B and Tencent is Hy-MT2 30B, both at full precision with a plain request. Plain E4B is stock Gemma 4 E4B at full precision, with a plain request plus "Reply with only the translation".

**Where the scores come from:** the 8 first languages use `judge_fresh2` (everyday texts), `judge_hard2` (difficult passages) and `judge_set2` (small-word texts); OpenTalk's 62 translations changed by the last fixes use `judge_changed`. The other 13 languages use `judge_l13`. So the first 8 languages were judged across several rounds, and the last 13 in one.

**Definitions:** serious = critical + major, excluding form. Critical, major, minor and cosmetic per 100 exclude form. Composite: 0 if unusable; otherwise 100 - (error points / length) - 5 x (5 - fluency) - 20 x (key items wrong / key items), not below 0, with error points of critical 40, major 20, minor 4, cosmetic 1, and length = source words / 50, at least 1. The composite is our own summary and has not been validated. "Double serious penalty" uses critical 80, major 40.

**Limits:**
- One AI judge, made by Haiku's maker. A Gemini round is running and is not in this report.
- Fixes were built from errors on the first 8 languages' everyday texts and difficult passages, and this version was chosen on them. The "Clean for OpenTalk" cut excludes those.
- Haiku's one overwritten translation counts as delivered, unusable. No system got a redo.
- The user's Persian overrulings are not applied.
- A language row is small, so single-language gaps are rough.

## Main cuts

### All 21 languages

3150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 5.0 | 3.5 | 4.1 | 11.5 | 13.4 | 13.0 |
| Translations with a serious error, % | 4.8 | 3.5 | 4.0 | 10.6 | 12.6 | 12.2 |
| Critical errors per 100 | 0.7 | 0.9 | 0.5 | 2.6 | 2.8 | 3.5 |
| Major errors per 100 | 4.3 | 2.6 | 3.6 | 8.9 | 10.6 | 9.5 |
| Minor errors per 100 | 9.6 | 6.1 | 8.9 | 16.5 | 14.6 | 18.9 |
| Cosmetic errors per 100 | 14.9 | 12.4 | 14.7 | 25.8 | 22.8 | 22.2 |
| All errors per 100 | 31.2 | 23.6 | 40.6 | 57.9 | 52.9 | 56.2 |
| Error-free translations, % | 72.7 | 78.4 | 63.9 | 53.3 | 56.7 | 53.8 |
| Fluency, mean of 1 to 5 | 4.39 | 4.62 | 4.26 | 4.33 | 4.46 | 4.11 |
| Fluency 5, % | 48.4 | 67.7 | 40.0 | 48.9 | 54.0 | 32.2 |
| Fluency 4, % | 42.5 | 27.0 | 46.4 | 37.4 | 38.5 | 48.3 |
| Fluency 3, % | 8.2 | 4.9 | 12.8 | 11.9 | 7.0 | 17.9 |
| Fluency 2 or 1, % | 0.9 | 0.3 | 0.8 | 1.8 | 0.5 | 1.6 |
| Key items wrong per 100 | 5.0 | 3.5 | 4.1 | 11.7 | 13.9 | 14.2 |
| Translations with a key item wrong, % | 4.8 | 3.5 | 3.9 | 10.3 | 12.8 | 12.5 |
| Share of key items wrong, % | 1.33 | 0.93 | 1.08 | 3.07 | 3.67 | 3.74 |
| Unusable, count | 1 | 3 | 0 | 27 | 4 | 8 |
| Unusable, % | 0.03 | 0.10 | 0.00 | 0.86 | 0.13 | 0.25 |
| Composite, mean | 95.29 | 96.82 | 94.80 | 92.21 | 92.99 | 90.95 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 89.00 | 92.44 | 89.00 | 78.68 | 76.42 | 73.33 |
| Composite, double serious penalty | 94.38 | 96.11 | 93.99 | 90.05 | 90.27 | 88.27 |
| Serious fidelity errors per 100 | 2.5 | 1.6 | 1.8 | 6.6 | 7.6 | 6.7 |
| Serious logic errors per 100 | 1.0 | 0.4 | 0.5 | 1.3 | 2.2 | 2.4 |
| Serious numbers errors per 100 | 0.9 | 1.1 | 1.5 | 2.3 | 3.0 | 3.0 |
| Serious comprehension errors per 100 | 0.6 | 0.3 | 0.3 | 1.3 | 0.6 | 1.0 |
| All fidelity errors per 100 | 10.3 | 8.0 | 6.8 | 25.6 | 26.9 | 17.9 |
| All logic errors per 100 | 2.6 | 1.7 | 1.4 | 3.9 | 5.4 | 3.9 |
| All numbers errors per 100 | 2.4 | 2.8 | 3.4 | 5.1 | 6.2 | 6.6 |
| All comprehension errors per 100 | 14.3 | 9.5 | 16.1 | 19.2 | 12.4 | 25.7 |
| All form errors per 100 | 1.7 | 1.6 | 12.9 | 4.1 | 2.0 | 2.2 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -1.5 (-3.1 to +0.2) | -0.9 (-2.4 to +0.7) | +6.5 (+4.3 to +9.0) clear | +8.3 (+5.8 to +11.3) clear | +8.0 (+5.8 to +10.4) clear |
| Critical errors per 100 | +0.3 (-0.5 to +1.2) | -0.2 (-0.6 to +0.3) | +2.0 (+0.8 to +3.6) clear | +2.1 (+0.9 to +3.8) clear | +2.8 (+1.6 to +4.4) clear |
| Error-free, points | +5.7 (+3.4 to +8.0) clear | -8.8 (-12.4 to -5.3) clear | -19.4 (-22.3 to -16.7) clear | -16.1 (-19.0 to -13.3) clear | -18.9 (-21.8 to -16.0) clear |
| Fluency | +0.23 (+0.20 to +0.27) clear | -0.13 (-0.18 to -0.08) clear | -0.05 (-0.09 to -0.01) clear | +0.08 (+0.03 to +0.12) clear | -0.27 (-0.32 to -0.23) clear |
| Key items wrong per 100 | -1.5 (-3.1 to +0.1) | -1.0 (-2.6 to +0.7) | +6.6 (+4.2 to +9.5) clear | +8.9 (+6.2 to +12.0) clear | +9.1 (+6.7 to +11.7) clear |
| Composite | +1.54 (+0.97 to +2.05) clear | -0.48 (-1.03 to +0.06) | -3.08 (-4.21 to -2.14) clear | -2.29 (-3.38 to -1.33) clear | -4.34 (-5.36 to -3.44) clear |

### Up to 40 words

1848 translations per system, from 88 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 3.7 | 2.7 | 4.6 | 11.2 | 12.9 | 12.9 |
| Translations with a serious error, % | 3.5 | 2.7 | 4.4 | 10.2 | 12.3 | 12.3 |
| Critical errors per 100 | 0.5 | 0.6 | 0.6 | 3.5 | 3.4 | 3.9 |
| Major errors per 100 | 3.2 | 2.1 | 4.0 | 7.7 | 9.5 | 9.0 |
| Minor errors per 100 | 6.0 | 4.3 | 6.7 | 10.8 | 9.6 | 13.8 |
| Cosmetic errors per 100 | 11.4 | 8.5 | 10.2 | 19.4 | 15.9 | 17.0 |
| All errors per 100 | 22.1 | 16.7 | 33.7 | 45.0 | 40.2 | 45.6 |
| Error-free translations, % | 79.4 | 84.1 | 68.9 | 60.8 | 64.7 | 60.3 |
| Fluency, mean of 1 to 5 | 4.46 | 4.65 | 4.32 | 4.35 | 4.48 | 4.18 |
| Fluency 5, % | 53.8 | 69.5 | 46.0 | 50.8 | 56.9 | 37.9 |
| Fluency 4, % | 39.1 | 25.8 | 41.2 | 35.5 | 35.3 | 44.0 |
| Fluency 3, % | 6.0 | 4.5 | 11.7 | 11.5 | 7.1 | 16.0 |
| Fluency 2 or 1, % | 1.1 | 0.2 | 1.0 | 2.3 | 0.6 | 2.1 |
| Key items wrong per 100 | 4.1 | 2.9 | 4.7 | 11.6 | 13.9 | 13.9 |
| Translations with a key item wrong, % | 3.8 | 2.8 | 4.5 | 9.8 | 12.8 | 12.4 |
| Share of key items wrong, % | 1.37 | 0.97 | 1.59 | 3.94 | 4.69 | 4.71 |
| Unusable, count | 0 | 0 | 0 | 15 | 3 | 7 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.81 | 0.16 | 0.38 |
| Composite, mean | 95.82 | 97.14 | 94.87 | 91.89 | 92.62 | 90.69 |
| Composite, median | 100.00 | 100.00 | 95.00 | 95.00 | 99.00 | 95.00 |
| Composite, lowest 10% cut-off | 91.00 | 94.00 | 89.00 | 73.33 | 73.33 | 68.33 |
| Composite, double serious penalty | 95.01 | 96.49 | 93.83 | 89.41 | 89.50 | 87.65 |
| Serious fidelity errors per 100 | 1.6 | 1.1 | 2.0 | 5.7 | 7.1 | 6.1 |
| Serious logic errors per 100 | 1.1 | 0.6 | 0.7 | 1.6 | 2.9 | 2.8 |
| Serious numbers errors per 100 | 0.7 | 0.8 | 1.7 | 2.6 | 2.3 | 3.0 |
| Serious comprehension errors per 100 | 0.3 | 0.2 | 0.2 | 1.2 | 0.7 | 1.0 |
| All fidelity errors per 100 | 6.9 | 5.2 | 4.9 | 17.4 | 19.2 | 14.1 |
| All logic errors per 100 | 2.6 | 1.5 | 1.8 | 3.8 | 5.5 | 4.5 |
| All numbers errors per 100 | 1.6 | 2.2 | 3.6 | 4.9 | 4.1 | 5.3 |
| All comprehension errors per 100 | 9.9 | 6.5 | 11.2 | 15.3 | 9.6 | 19.9 |
| All form errors per 100 | 1.1 | 1.2 | 12.2 | 3.5 | 1.8 | 1.9 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -1.1 (-2.8 to +0.3) | +0.9 (-1.2 to +2.9) | +7.5 (+4.3 to +11.0) clear | +9.2 (+5.2 to +13.7) clear | +9.2 (+5.9 to +12.9) clear |
| Critical errors per 100 | +0.1 (-0.4 to +0.6) | +0.2 (-0.3 to +0.6) | +3.0 (+1.1 to +5.6) clear | +2.9 (+0.9 to +5.5) clear | +3.4 (+1.5 to +6.0) clear |
| Error-free, points | +4.7 (+2.1 to +7.4) clear | -10.6 (-15.4 to -5.8) clear | -18.7 (-22.3 to -15.0) clear | -14.7 (-19.0 to -10.7) clear | -19.1 (-23.1 to -15.3) clear |
| Fluency | +0.19 (+0.14 to +0.23) clear | -0.13 (-0.20 to -0.07) clear | -0.11 (-0.17 to -0.06) clear | +0.03 (-0.03 to +0.08) | -0.28 (-0.34 to -0.22) clear |
| Key items wrong per 100 | -1.2 (-3.1 to +0.4) | +0.6 (-1.6 to +2.8) | +7.6 (+3.9 to +12.1) clear | +9.8 (+5.4 to +14.7) clear | +9.8 (+6.1 to +13.9) clear |
| Composite | +1.32 (+0.76 to +1.94) clear | -0.95 (-1.69 to -0.20) clear | -3.93 (-5.55 to -2.57) clear | -3.20 (-4.93 to -1.68) clear | -5.13 (-6.71 to -3.74) clear |

### Over 40 words

1302 translations per system, from 62 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 6.8 | 4.7 | 3.4 | 11.9 | 14.0 | 13.1 |
| Translations with a serious error, % | 6.6 | 4.6 | 3.3 | 11.1 | 13.1 | 12.1 |
| Critical errors per 100 | 0.9 | 1.4 | 0.2 | 1.5 | 1.9 | 2.9 |
| Major errors per 100 | 5.9 | 3.3 | 3.1 | 10.4 | 12.1 | 10.2 |
| Minor errors per 100 | 14.7 | 8.6 | 12.1 | 24.6 | 21.8 | 26.0 |
| Cosmetic errors per 100 | 19.8 | 17.9 | 21.2 | 34.9 | 32.7 | 29.6 |
| All errors per 100 | 44.0 | 33.4 | 50.5 | 76.2 | 70.8 | 71.3 |
| Error-free translations, % | 63.2 | 70.4 | 56.8 | 42.8 | 45.2 | 44.6 |
| Fluency, mean of 1 to 5 | 4.29 | 4.58 | 4.16 | 4.31 | 4.43 | 4.02 |
| Fluency 5, % | 40.8 | 65.1 | 31.5 | 46.2 | 50.0 | 24.0 |
| Fluency 4, % | 47.5 | 28.8 | 53.7 | 40.2 | 43.1 | 54.5 |
| Fluency 3, % | 11.3 | 5.5 | 14.3 | 12.5 | 6.7 | 20.6 |
| Fluency 2 or 1, % | 0.5 | 0.5 | 0.5 | 1.2 | 0.2 | 0.9 |
| Key items wrong per 100 | 6.5 | 4.5 | 3.2 | 11.7 | 14.0 | 14.6 |
| Translations with a key item wrong, % | 6.2 | 4.5 | 3.1 | 10.8 | 12.9 | 12.7 |
| Share of key items wrong, % | 1.29 | 0.89 | 0.65 | 2.34 | 2.80 | 2.93 |
| Unusable, count | 1 | 3 | 0 | 12 | 1 | 1 |
| Unusable, % | 0.08 | 0.23 | 0.00 | 0.92 | 0.08 | 0.08 |
| Composite, mean | 94.53 | 96.38 | 94.71 | 92.66 | 93.53 | 91.32 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 94.38 |
| Composite, lowest 10% cut-off | 87.22 | 90.00 | 88.53 | 80.29 | 81.29 | 77.54 |
| Composite, double serious penalty | 93.48 | 95.58 | 94.21 | 90.95 | 91.37 | 89.15 |
| Serious fidelity errors per 100 | 3.8 | 2.2 | 1.5 | 7.8 | 8.2 | 7.5 |
| Serious logic errors per 100 | 0.9 | 0.2 | 0.3 | 0.8 | 1.2 | 1.8 |
| Serious numbers errors per 100 | 1.2 | 1.6 | 1.2 | 1.8 | 4.1 | 2.9 |
| Serious comprehension errors per 100 | 0.9 | 0.6 | 0.4 | 1.5 | 0.4 | 0.9 |
| All fidelity errors per 100 | 15.0 | 12.0 | 9.4 | 37.2 | 37.7 | 23.3 |
| All logic errors per 100 | 2.5 | 1.9 | 0.9 | 4.1 | 5.2 | 3.1 |
| All numbers errors per 100 | 3.5 | 3.7 | 3.1 | 5.3 | 9.2 | 8.5 |
| All comprehension errors per 100 | 20.4 | 13.6 | 23.1 | 24.8 | 16.4 | 33.9 |
| All form errors per 100 | 2.6 | 2.2 | 13.8 | 4.8 | 2.3 | 2.5 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -2.2 (-5.0 to +1.2) | -3.5 (-5.5 to -1.2) clear | +5.1 (+2.3 to +8.0) clear | +7.1 (+3.7 to +11.1) clear | +6.3 (+3.8 to +9.4) clear |
| Critical errors per 100 | +0.5 (-1.1 to +2.5) | -0.7 (-1.5 to +0.1) | +0.5 (-0.5 to +1.6) | +1.0 (-0.2 to +2.3) | +2.0 (+0.7 to +3.4) clear |
| Error-free, points | +7.2 (+3.4 to +10.9) clear | -6.4 (-11.8 to -0.8) clear | -20.4 (-25.1 to -16.0) clear | -18.0 (-21.9 to -14.1) clear | -18.6 (-22.8 to -14.4) clear |
| Fluency | +0.30 (+0.25 to +0.35) clear | -0.12 (-0.20 to -0.05) clear | +0.03 (-0.02 to +0.08) | +0.14 (+0.09 to +0.19) clear | -0.27 (-0.32 to -0.22) clear |
| Key items wrong per 100 | -2.0 (-4.6 to +1.2) | -3.2 (-5.4 to -1.1) clear | +5.2 (+2.5 to +8.1) clear | +7.5 (+4.4 to +10.9) clear | +8.1 (+5.4 to +11.2) clear |
| Composite | +1.85 (+0.83 to +2.70) clear | +0.18 (-0.54 to +0.92) | -1.86 (-2.85 to -0.97) clear | -1.00 (-1.98 to -0.12) clear | -3.20 (-4.04 to -2.42) clear |

### Up to 25 words

1302 translations per system, from 62 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 2.7 | 2.1 | 4.5 | 9.5 | 11.0 | 12.3 |
| Translations with a serious error, % | 2.5 | 2.1 | 4.4 | 8.6 | 10.5 | 11.8 |
| Critical errors per 100 | 0.4 | 0.4 | 0.6 | 3.1 | 2.5 | 4.1 |
| Major errors per 100 | 2.3 | 1.7 | 3.8 | 6.5 | 8.5 | 8.1 |
| Minor errors per 100 | 4.5 | 3.4 | 5.3 | 9.4 | 7.8 | 12.2 |
| Cosmetic errors per 100 | 10.5 | 8.0 | 10.1 | 17.9 | 14.2 | 16.1 |
| All errors per 100 | 19.0 | 14.7 | 30.4 | 40.4 | 34.4 | 42.1 |
| Error-free translations, % | 81.9 | 85.5 | 70.9 | 63.7 | 68.2 | 62.1 |
| Fluency, mean of 1 to 5 | 4.49 | 4.69 | 4.38 | 4.38 | 4.50 | 4.23 |
| Fluency 5, % | 56.6 | 72.4 | 49.8 | 53.3 | 59.3 | 41.9 |
| Fluency 4, % | 37.2 | 24.0 | 39.6 | 33.6 | 32.6 | 40.9 |
| Fluency 3, % | 5.3 | 3.5 | 9.6 | 10.8 | 7.3 | 15.4 |
| Fluency 2 or 1, % | 0.9 | 0.2 | 1.1 | 2.2 | 0.8 | 1.8 |
| Key items wrong per 100 | 2.9 | 2.2 | 4.5 | 10.4 | 11.5 | 13.2 |
| Translations with a key item wrong, % | 2.8 | 2.2 | 4.3 | 8.4 | 10.9 | 11.8 |
| Share of key items wrong, % | 1.04 | 0.79 | 1.59 | 3.69 | 4.11 | 4.71 |
| Unusable, count | 0 | 0 | 0 | 12 | 2 | 3 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.92 | 0.15 | 0.23 |
| Composite, mean | 96.35 | 97.56 | 95.24 | 92.54 | 93.47 | 91.12 |
| Composite, median | 100.00 | 100.00 | 95.00 | 99.00 | 100.00 | 95.00 |
| Composite, lowest 10% cut-off | 94.00 | 94.00 | 89.00 | 81.00 | 73.33 | 68.33 |
| Composite, double serious penalty | 95.78 | 97.07 | 94.24 | 90.49 | 90.92 | 88.16 |
| Serious fidelity errors per 100 | 1.5 | 0.7 | 2.1 | 4.6 | 6.1 | 5.5 |
| Serious logic errors per 100 | 0.2 | 0.2 | 0.2 | 0.8 | 1.4 | 2.3 |
| Serious numbers errors per 100 | 0.7 | 1.0 | 2.0 | 2.8 | 2.5 | 3.5 |
| Serious comprehension errors per 100 | 0.2 | 0.2 | 0.2 | 1.2 | 0.9 | 1.1 |
| All fidelity errors per 100 | 6.5 | 4.3 | 4.7 | 14.7 | 15.7 | 12.8 |
| All logic errors per 100 | 1.1 | 0.8 | 0.8 | 2.2 | 3.4 | 3.3 |
| All numbers errors per 100 | 1.5 | 2.5 | 4.2 | 5.5 | 4.4 | 5.5 |
| All comprehension errors per 100 | 8.6 | 5.8 | 10.1 | 14.5 | 9.5 | 18.9 |
| All form errors per 100 | 1.3 | 1.3 | 10.6 | 3.6 | 1.5 | 1.5 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -0.6 (-2.1 to +0.8) | +1.8 (-0.6 to +4.3) | +6.8 (+3.4 to +11.5) clear | +8.3 (+4.5 to +12.9) clear | +9.6 (+5.8 to +13.7) clear |
| Critical errors per 100 | +0.0 (-0.5 to +0.5) | +0.2 (-0.3 to +0.8) | +2.7 (+0.5 to +6.1) clear | +2.1 (+0.8 to +3.7) clear | +3.8 (+1.3 to +7.2) clear |
| Error-free, points | +3.6 (+0.5 to +6.6) clear | -11.0 (-16.5 to -5.3) clear | -18.1 (-22.7 to -13.9) clear | -13.7 (-18.3 to -9.4) clear | -19.7 (-24.3 to -15.4) clear |
| Fluency | +0.19 (+0.14 to +0.24) clear | -0.11 (-0.20 to -0.04) clear | -0.12 (-0.19 to -0.05) clear | +0.01 (-0.05 to +0.07) | -0.27 (-0.34 to -0.20) clear |
| Key items wrong per 100 | -0.7 (-2.3 to +0.8) | +1.5 (-0.8 to +4.2) | +7.5 (+3.1 to +13.7) clear | +8.6 (+4.6 to +13.4) clear | +10.3 (+6.1 to +15.1) clear |
| Composite | +1.20 (+0.63 to +1.80) clear | -1.11 (-2.06 to -0.24) clear | -3.82 (-6.18 to -2.07) clear | -2.88 (-4.43 to -1.56) clear | -5.23 (-7.17 to -3.44) clear |

### 26 to 40 words

546 translations per system, from 26 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 6.2 | 4.0 | 4.9 | 15.2 | 17.6 | 14.5 |
| Translations with a serious error, % | 6.0 | 4.0 | 4.6 | 14.1 | 16.5 | 13.4 |
| Critical errors per 100 | 0.7 | 1.1 | 0.7 | 4.4 | 5.7 | 3.3 |
| Major errors per 100 | 5.5 | 2.9 | 4.2 | 10.8 | 11.9 | 11.2 |
| Minor errors per 100 | 9.3 | 6.4 | 10.1 | 14.3 | 13.9 | 17.6 |
| Cosmetic errors per 100 | 13.4 | 9.9 | 10.4 | 23.1 | 19.8 | 19.2 |
| All errors per 100 | 29.5 | 21.2 | 41.6 | 55.9 | 54.0 | 54.0 |
| Error-free translations, % | 73.6 | 80.8 | 64.1 | 53.7 | 56.4 | 56.0 |
| Fluency, mean of 1 to 5 | 4.36 | 4.55 | 4.19 | 4.27 | 4.44 | 4.06 |
| Fluency 5, % | 47.3 | 62.8 | 37.2 | 44.7 | 51.1 | 28.6 |
| Fluency 4, % | 43.6 | 30.0 | 45.2 | 39.9 | 41.8 | 51.3 |
| Fluency 3, % | 7.5 | 6.8 | 16.7 | 13.0 | 6.8 | 17.4 |
| Fluency 2 or 1, % | 1.6 | 0.4 | 0.9 | 2.4 | 0.4 | 2.7 |
| Key items wrong per 100 | 6.8 | 4.4 | 5.3 | 14.7 | 19.4 | 15.6 |
| Translations with a key item wrong, % | 6.4 | 4.2 | 4.9 | 13.4 | 17.2 | 13.7 |
| Share of key items wrong, % | 2.05 | 1.33 | 1.61 | 4.43 | 5.87 | 4.71 |
| Unusable, count | 0 | 0 | 0 | 3 | 1 | 4 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.55 | 0.18 | 0.73 |
| Composite, mean | 94.55 | 96.14 | 93.99 | 90.35 | 90.59 | 89.65 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 86.00 | 90.00 | 86.00 | 68.33 | 68.33 | 68.33 |
| Composite, double serious penalty | 93.17 | 95.11 | 92.85 | 86.83 | 86.11 | 86.42 |
| Serious fidelity errors per 100 | 1.6 | 2.0 | 1.8 | 8.4 | 9.3 | 7.7 |
| Serious logic errors per 100 | 3.3 | 1.5 | 1.8 | 3.5 | 6.4 | 3.8 |
| Serious numbers errors per 100 | 0.7 | 0.4 | 0.9 | 2.0 | 1.6 | 2.0 |
| Serious comprehension errors per 100 | 0.5 | 0.2 | 0.4 | 1.3 | 0.2 | 0.9 |
| All fidelity errors per 100 | 7.9 | 7.5 | 5.5 | 23.8 | 27.7 | 17.0 |
| All logic errors per 100 | 6.2 | 3.3 | 4.0 | 7.9 | 10.6 | 7.3 |
| All numbers errors per 100 | 1.8 | 1.3 | 2.0 | 3.7 | 3.3 | 4.8 |
| All comprehension errors per 100 | 13.0 | 8.2 | 13.9 | 17.2 | 9.7 | 22.2 |
| All form errors per 100 | 0.5 | 0.9 | 16.1 | 3.3 | 2.7 | 2.7 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -2.2 (-6.8 to +1.8) | -1.3 (-5.9 to +2.2) | +9.0 (+3.3 to +15.8) clear | +11.4 (+3.1 to +21.8) clear | +8.2 (+1.5 to +15.8) clear |
| Critical errors per 100 | +0.4 (-0.5 to +1.8) | +0.0 (-0.7 to +0.9) | +3.7 (+1.1 to +7.3) clear | +4.9 (-0.5 to +12.8) | +2.6 (+0.5 to +5.1) clear |
| Error-free, points | +7.1 (+1.5 to +12.8) clear | -9.5 (-16.8 to -2.0) clear | -20.0 (-27.5 to -13.7) clear | -17.2 (-26.4 to -8.6) clear | -17.6 (-25.6 to -10.6) clear |
| Fluency | +0.19 (+0.10 to +0.29) clear | -0.18 (-0.29 to -0.06) clear | -0.10 (-0.19 to +0.00) | +0.07 (-0.05 to +0.19) | -0.31 (-0.40 to -0.22) clear |
| Key items wrong per 100 | -2.4 (-7.5 to +2.0) | -1.5 (-6.6 to +2.6) | +7.9 (+2.0 to +15.0) clear | +12.6 (+3.5 to +24.2) clear | +8.8 (+1.1 to +17.0) clear |
| Composite | +1.59 (+0.13 to +3.13) clear | -0.56 (-1.70 to +0.69) | -4.20 (-7.00 to -2.00) clear | -3.96 (-8.39 to -0.42) clear | -4.90 (-7.51 to -2.61) clear |

### 41 to 60 words

21 translations per system, from 1 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 9.5 | 0.0 | 4.8 | 0.0 | 4.8 | 4.8 |
| Translations with a serious error, % | 9.5 | 0.0 | 4.8 | 0.0 | 4.8 | 4.8 |
| Critical errors per 100 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Major errors per 100 | 9.5 | 0.0 | 4.8 | 0.0 | 4.8 | 4.8 |
| Minor errors per 100 | 4.8 | 0.0 | 14.3 | 9.5 | 14.3 | 9.5 |
| Cosmetic errors per 100 | 14.3 | 14.3 | 9.5 | 28.6 | 23.8 | 14.3 |
| All errors per 100 | 28.6 | 19.0 | 33.3 | 42.9 | 42.9 | 28.6 |
| Error-free translations, % | 71.4 | 81.0 | 66.7 | 61.9 | 61.9 | 76.2 |
| Fluency, mean of 1 to 5 | 4.38 | 4.57 | 4.29 | 4.48 | 4.43 | 4.29 |
| Fluency 5, % | 47.6 | 61.9 | 42.9 | 52.4 | 47.6 | 33.3 |
| Fluency 4, % | 42.9 | 33.3 | 42.9 | 42.9 | 47.6 | 61.9 |
| Fluency 3, % | 9.5 | 4.8 | 14.3 | 4.8 | 4.8 | 4.8 |
| Fluency 2 or 1, % | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Key items wrong per 100 | 9.5 | 0.0 | 4.8 | 0.0 | 4.8 | 4.8 |
| Translations with a key item wrong, % | 9.5 | 0.0 | 4.8 | 0.0 | 4.8 | 4.8 |
| Share of key items wrong, % | 2.38 | 0.00 | 1.19 | 0.00 | 1.19 | 1.19 |
| Unusable, count | 0 | 0 | 0 | 0 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Composite, mean | 94.53 | 97.74 | 94.82 | 96.82 | 95.41 | 94.94 |
| Composite, median | 95.00 | 100.00 | 95.00 | 96.61 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 86.61 | 94.15 | 86.61 | 94.15 | 91.61 | 91.61 |
| Composite, double serious penalty | 92.92 | 97.74 | 94.01 | 96.82 | 94.60 | 94.13 |
| Serious fidelity errors per 100 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Serious logic errors per 100 | 4.8 | 0.0 | 4.8 | 0.0 | 4.8 | 4.8 |
| Serious numbers errors per 100 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Serious comprehension errors per 100 | 4.8 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| All fidelity errors per 100 | 4.8 | 4.8 | 4.8 | 33.3 | 14.3 | 4.8 |
| All logic errors per 100 | 4.8 | 0.0 | 9.5 | 0.0 | 9.5 | 9.5 |
| All numbers errors per 100 | 0.0 | 4.8 | 4.8 | 0.0 | 0.0 | 0.0 |
| All comprehension errors per 100 | 19.0 | 4.8 | 9.5 | 4.8 | 19.0 | 14.3 |
| All form errors per 100 | 0.0 | 4.8 | 4.8 | 4.8 | 0.0 | 0.0 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -9.5 (-9.5 to -9.5) clear | -4.8 (-4.8 to -4.8) clear | -9.5 (-9.5 to -9.5) clear | -4.8 (-4.8 to -4.8) clear | -4.8 (-4.8 to -4.8) clear |
| Critical errors per 100 | +0.0 (+0.0 to +0.0) | +0.0 (+0.0 to +0.0) | +0.0 (+0.0 to +0.0) | +0.0 (+0.0 to +0.0) | +0.0 (+0.0 to +0.0) |
| Error-free, points | +9.5 (+9.5 to +9.5) clear | -4.8 (-4.8 to -4.8) clear | -9.5 (-9.5 to -9.5) clear | -9.5 (-9.5 to -9.5) clear | +4.8 (+4.8 to +4.8) clear |
| Fluency | +0.19 (+0.19 to +0.19) clear | -0.10 (-0.10 to -0.10) clear | +0.10 (+0.10 to +0.10) clear | +0.05 (+0.05 to +0.05) clear | -0.10 (-0.10 to -0.10) clear |
| Key items wrong per 100 | -9.5 (-9.5 to -9.5) clear | -4.8 (-4.8 to -4.8) clear | -9.5 (-9.5 to -9.5) clear | -4.8 (-4.8 to -4.8) clear | -4.8 (-4.8 to -4.8) clear |
| Composite | +3.20 (+3.20 to +3.20) clear | +0.29 (+0.29 to +0.29) clear | +2.28 (+2.28 to +2.28) clear | +0.88 (+0.88 to +0.88) clear | +0.41 (+0.41 to +0.41) clear |

### Over 60 words

1281 translations per system, from 61 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 6.8 | 4.8 | 3.4 | 12.1 | 14.1 | 13.3 |
| Translations with a serious error, % | 6.6 | 4.7 | 3.3 | 11.3 | 13.3 | 12.3 |
| Critical errors per 100 | 0.9 | 1.4 | 0.2 | 1.5 | 2.0 | 3.0 |
| Major errors per 100 | 5.9 | 3.4 | 3.1 | 10.6 | 12.2 | 10.3 |
| Minor errors per 100 | 14.9 | 8.7 | 12.0 | 24.8 | 21.9 | 26.3 |
| Cosmetic errors per 100 | 19.9 | 18.0 | 21.4 | 35.0 | 32.9 | 29.8 |
| All errors per 100 | 44.3 | 33.6 | 50.7 | 76.7 | 71.3 | 72.0 |
| Error-free translations, % | 63.1 | 70.3 | 56.7 | 42.5 | 45.0 | 44.1 |
| Fluency, mean of 1 to 5 | 4.28 | 4.58 | 4.16 | 4.31 | 4.43 | 4.01 |
| Fluency 5, % | 40.7 | 65.2 | 31.3 | 46.1 | 50.0 | 23.9 |
| Fluency 4, % | 47.5 | 28.7 | 53.9 | 40.1 | 43.0 | 54.3 |
| Fluency 3, % | 11.3 | 5.5 | 14.3 | 12.6 | 6.7 | 20.8 |
| Fluency 2 or 1, % | 0.5 | 0.5 | 0.5 | 1.2 | 0.2 | 0.9 |
| Key items wrong per 100 | 6.4 | 4.5 | 3.2 | 11.9 | 14.1 | 14.8 |
| Translations with a key item wrong, % | 6.2 | 4.5 | 3.1 | 11.0 | 13.0 | 12.9 |
| Share of key items wrong, % | 1.28 | 0.91 | 0.64 | 2.37 | 2.83 | 2.95 |
| Unusable, count | 1 | 3 | 0 | 12 | 1 | 1 |
| Unusable, % | 0.08 | 0.23 | 0.00 | 0.94 | 0.08 | 0.08 |
| Composite, mean | 94.53 | 96.36 | 94.70 | 92.60 | 93.50 | 91.26 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 94.38 |
| Composite, lowest 10% cut-off | 87.26 | 90.00 | 88.78 | 80.29 | 80.95 | 77.35 |
| Composite, double serious penalty | 93.49 | 95.54 | 94.22 | 90.85 | 91.32 | 89.07 |
| Serious fidelity errors per 100 | 3.9 | 2.3 | 1.6 | 8.0 | 8.4 | 7.6 |
| Serious logic errors per 100 | 0.9 | 0.2 | 0.2 | 0.9 | 1.2 | 1.8 |
| Serious numbers errors per 100 | 1.2 | 1.6 | 1.2 | 1.8 | 4.2 | 3.0 |
| Serious comprehension errors per 100 | 0.9 | 0.6 | 0.4 | 1.5 | 0.4 | 0.9 |
| All fidelity errors per 100 | 15.1 | 12.1 | 9.5 | 37.2 | 38.1 | 23.6 |
| All logic errors per 100 | 2.5 | 2.0 | 0.8 | 4.1 | 5.2 | 3.0 |
| All numbers errors per 100 | 3.5 | 3.7 | 3.1 | 5.4 | 9.4 | 8.7 |
| All comprehension errors per 100 | 20.5 | 13.7 | 23.3 | 25.1 | 16.3 | 34.2 |
| All form errors per 100 | 2.7 | 2.2 | 14.0 | 4.8 | 2.3 | 2.6 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -2.0 (-4.8 to +1.4) | -3.4 (-5.5 to -1.2) clear | +5.3 (+2.4 to +8.4) clear | +7.3 (+3.5 to +11.4) clear | +6.5 (+4.0 to +9.1) clear |
| Critical errors per 100 | +0.5 (-1.1 to +2.6) | -0.7 (-1.6 to +0.1) | +0.5 (-0.5 to +1.6) | +1.0 (-0.2 to +2.3) | +2.0 (+0.7 to +3.4) clear |
| Error-free, points | +7.2 (+3.3 to +11.2) clear | -6.4 (-12.3 to -0.6) clear | -20.6 (-25.4 to -15.9) clear | -18.1 (-22.2 to -14.1) clear | -19.0 (-23.2 to -14.8) clear |
| Fluency | +0.30 (+0.25 to +0.35) clear | -0.12 (-0.20 to -0.05) clear | +0.03 (-0.02 to +0.08) | +0.14 (+0.09 to +0.20) clear | -0.27 (-0.32 to -0.22) clear |
| Key items wrong per 100 | -1.9 (-4.6 to +1.3) | -3.2 (-5.4 to -1.0) clear | +5.5 (+2.7 to +8.2) clear | +7.7 (+4.2 to +11.2) clear | +8.4 (+5.6 to +11.2) clear |
| Composite | +1.83 (+0.77 to +2.72) clear | +0.18 (-0.59 to +0.96) | -1.93 (-2.89 to -0.98) clear | -1.03 (-1.94 to -0.04) clear | -3.26 (-4.05 to -2.44) clear |

### 16 main languages

2400 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 4.3 | 3.4 | 3.3 | 10.4 | 12.6 | 12.0 |
| Translations with a serious error, % | 4.2 | 3.4 | 3.2 | 9.6 | 11.9 | 11.3 |
| Critical errors per 100 | 0.6 | 1.0 | 0.4 | 2.3 | 2.6 | 3.5 |
| Major errors per 100 | 3.7 | 2.4 | 2.9 | 8.1 | 10.0 | 8.5 |
| Minor errors per 100 | 9.0 | 5.3 | 8.6 | 14.5 | 15.6 | 17.5 |
| Cosmetic errors per 100 | 13.6 | 11.2 | 14.6 | 25.2 | 23.9 | 21.4 |
| All errors per 100 | 28.4 | 21.5 | 39.9 | 54.3 | 54.3 | 53.0 |
| Error-free translations, % | 75.2 | 80.0 | 64.7 | 55.3 | 55.6 | 55.8 |
| Fluency, mean of 1 to 5 | 4.40 | 4.64 | 4.27 | 4.37 | 4.46 | 4.13 |
| Fluency 5, % | 49.6 | 68.8 | 41.2 | 51.7 | 53.8 | 32.9 |
| Fluency 4, % | 41.9 | 26.4 | 44.8 | 35.5 | 38.5 | 48.7 |
| Fluency 3, % | 7.8 | 4.5 | 13.3 | 11.1 | 7.1 | 17.2 |
| Fluency 2 or 1, % | 0.7 | 0.3 | 0.7 | 1.7 | 0.5 | 1.2 |
| Key items wrong per 100 | 4.3 | 3.5 | 3.4 | 10.5 | 13.1 | 13.0 |
| Translations with a key item wrong, % | 4.1 | 3.4 | 3.3 | 9.2 | 12.1 | 11.5 |
| Share of key items wrong, % | 1.14 | 0.92 | 0.90 | 2.78 | 3.46 | 3.42 |
| Unusable, count | 1 | 3 | 0 | 23 | 3 | 6 |
| Unusable, % | 0.04 | 0.12 | 0.00 | 0.96 | 0.12 | 0.25 |
| Composite, mean | 95.57 | 96.93 | 95.08 | 92.71 | 93.16 | 91.38 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 89.24 | 94.00 | 89.00 | 81.00 | 77.30 | 76.00 |
| Composite, double serious penalty | 94.79 | 96.23 | 94.43 | 90.76 | 90.59 | 88.89 |
| Serious fidelity errors per 100 | 2.2 | 1.5 | 1.5 | 5.9 | 7.6 | 6.7 |
| Serious logic errors per 100 | 0.8 | 0.4 | 0.4 | 1.0 | 1.8 | 2.0 |
| Serious numbers errors per 100 | 0.7 | 1.1 | 1.1 | 2.2 | 2.6 | 2.5 |
| Serious comprehension errors per 100 | 0.6 | 0.4 | 0.2 | 1.2 | 0.6 | 0.8 |
| All fidelity errors per 100 | 9.9 | 7.0 | 6.5 | 24.5 | 27.6 | 17.0 |
| All logic errors per 100 | 2.2 | 1.7 | 1.1 | 3.6 | 5.4 | 3.4 |
| All numbers errors per 100 | 2.2 | 2.6 | 3.0 | 4.7 | 5.8 | 6.0 |
| All comprehension errors per 100 | 12.7 | 8.5 | 15.8 | 17.2 | 13.2 | 24.5 |
| All form errors per 100 | 1.4 | 1.7 | 13.3 | 4.3 | 2.2 | 2.0 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -1.0 (-2.4 to +0.7) | -1.0 (-2.5 to +0.4) | +6.0 (+4.0 to +8.5) clear | +8.3 (+5.7 to +11.3) clear | +7.7 (+5.5 to +10.1) clear |
| Critical errors per 100 | +0.3 (-0.5 to +1.2) | -0.3 (-0.7 to +0.2) | +1.7 (+0.5 to +3.2) clear | +2.0 (+0.8 to +3.5) clear | +2.9 (+1.7 to +4.5) clear |
| Error-free, points | +4.7 (+2.3 to +7.1) clear | -10.5 (-14.5 to -6.8) clear | -20.0 (-23.0 to -17.1) clear | -19.6 (-22.8 to -16.4) clear | -19.5 (-22.5 to -16.5) clear |
| Fluency | +0.23 (+0.19 to +0.27) clear | -0.14 (-0.19 to -0.08) clear | -0.03 (-0.08 to +0.01) | +0.05 (+0.01 to +0.10) clear | -0.27 (-0.31 to -0.23) clear |
| Key items wrong per 100 | -0.8 (-2.4 to +0.8) | -0.9 (-2.4 to +0.7) | +6.2 (+3.8 to +9.3) clear | +8.8 (+6.1 to +11.9) clear | +8.6 (+6.3 to +11.2) clear |
| Composite | +1.36 (+0.79 to +1.91) clear | -0.50 (-1.04 to +0.07) | -2.87 (-4.00 to -1.91) clear | -2.41 (-3.46 to -1.47) clear | -4.20 (-5.19 to -3.30) clear |

### 16 main languages, up to 40 words

1408 translations per system, from 88 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 2.9 | 2.6 | 3.6 | 9.7 | 12.0 | 11.9 |
| Translations with a serious error, % | 2.7 | 2.6 | 3.4 | 8.9 | 11.4 | 11.2 |
| Critical errors per 100 | 0.5 | 0.6 | 0.5 | 3.1 | 3.1 | 3.8 |
| Major errors per 100 | 2.4 | 2.0 | 3.1 | 6.6 | 8.9 | 8.0 |
| Minor errors per 100 | 5.6 | 4.1 | 6.5 | 9.7 | 10.1 | 13.4 |
| Cosmetic errors per 100 | 10.2 | 8.5 | 10.1 | 19.4 | 16.4 | 16.6 |
| All errors per 100 | 19.7 | 16.5 | 32.5 | 42.3 | 40.6 | 43.6 |
| Error-free translations, % | 81.9 | 84.3 | 69.8 | 62.6 | 64.3 | 61.8 |
| Fluency, mean of 1 to 5 | 4.48 | 4.66 | 4.34 | 4.38 | 4.48 | 4.19 |
| Fluency 5, % | 55.3 | 70.5 | 47.6 | 53.0 | 56.7 | 38.4 |
| Fluency 4, % | 38.1 | 24.7 | 39.6 | 34.3 | 35.5 | 44.0 |
| Fluency 3, % | 5.8 | 4.5 | 12.1 | 10.7 | 7.2 | 16.1 |
| Fluency 2 or 1, % | 0.9 | 0.2 | 0.7 | 2.1 | 0.6 | 1.6 |
| Key items wrong per 100 | 3.2 | 2.8 | 3.7 | 10.4 | 13.0 | 12.6 |
| Translations with a key item wrong, % | 3.0 | 2.7 | 3.6 | 8.5 | 12.1 | 11.3 |
| Share of key items wrong, % | 1.08 | 0.96 | 1.25 | 3.51 | 4.40 | 4.25 |
| Unusable, count | 0 | 0 | 0 | 13 | 2 | 5 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.92 | 0.14 | 0.36 |
| Composite, mean | 96.17 | 97.20 | 95.28 | 92.46 | 92.91 | 91.14 |
| Composite, median | 100.00 | 100.00 | 95.00 | 96.00 | 99.00 | 95.00 |
| Composite, lowest 10% cut-off | 91.00 | 94.00 | 89.00 | 81.00 | 73.33 | 70.00 |
| Composite, double serious penalty | 95.50 | 96.54 | 94.47 | 90.28 | 89.99 | 88.35 |
| Serious fidelity errors per 100 | 1.3 | 1.2 | 1.5 | 4.7 | 7.1 | 6.1 |
| Serious logic errors per 100 | 0.9 | 0.6 | 0.6 | 1.3 | 2.3 | 2.4 |
| Serious numbers errors per 100 | 0.4 | 0.6 | 1.3 | 2.5 | 1.8 | 2.6 |
| Serious comprehension errors per 100 | 0.4 | 0.2 | 0.1 | 1.2 | 0.8 | 0.8 |
| All fidelity errors per 100 | 6.6 | 5.2 | 4.7 | 16.7 | 19.7 | 13.9 |
| All logic errors per 100 | 2.3 | 1.6 | 1.4 | 3.6 | 5.4 | 4.0 |
| All numbers errors per 100 | 1.1 | 2.0 | 3.3 | 4.7 | 3.5 | 4.8 |
| All comprehension errors per 100 | 8.7 | 6.5 | 10.7 | 13.8 | 9.9 | 19.2 |
| All form errors per 100 | 1.0 | 1.3 | 12.4 | 3.6 | 2.1 | 1.7 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -0.3 (-1.8 to +1.2) | +0.6 (-1.1 to +2.2) | +6.7 (+3.8 to +10.4) clear | +9.1 (+5.4 to +13.5) clear | +8.9 (+5.9 to +12.4) clear |
| Critical errors per 100 | +0.1 (-0.4 to +0.8) | +0.0 (-0.6 to +0.6) | +2.6 (+0.9 to +5.1) clear | +2.6 (+0.8 to +4.9) clear | +3.3 (+1.3 to +6.0) clear |
| Error-free, points | +2.4 (-0.2 to +5.2) | -12.1 (-16.9 to -7.1) clear | -19.3 (-23.2 to -15.4) clear | -17.5 (-22.4 to -13.1) clear | -20.1 (-24.3 to -16.1) clear |
| Fluency | +0.18 (+0.12 to +0.22) clear | -0.14 (-0.21 to -0.07) clear | -0.10 (-0.16 to -0.04) clear | +0.00 (-0.06 to +0.06) | -0.29 (-0.35 to -0.23) clear |
| Key items wrong per 100 | -0.4 (-2.2 to +1.3) | +0.5 (-1.4 to +2.3) | +7.2 (+3.5 to +12.1) clear | +9.8 (+5.8 to +14.6) clear | +9.4 (+6.0 to +13.1) clear |
| Composite | +1.03 (+0.40 to +1.65) clear | -0.89 (-1.55 to -0.15) clear | -3.71 (-5.41 to -2.38) clear | -3.26 (-4.90 to -1.87) clear | -5.03 (-6.64 to -3.68) clear |

### 16 main languages, over 40 words

992 translations per system, from 62 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 6.4 | 4.4 | 2.9 | 11.4 | 13.5 | 12.3 |
| Translations with a serious error, % | 6.2 | 4.4 | 2.8 | 10.6 | 12.6 | 11.4 |
| Critical errors per 100 | 0.8 | 1.4 | 0.2 | 1.2 | 1.9 | 3.0 |
| Major errors per 100 | 5.5 | 3.0 | 2.7 | 10.2 | 11.6 | 9.3 |
| Minor errors per 100 | 13.8 | 7.0 | 11.7 | 21.3 | 23.4 | 23.4 |
| Cosmetic errors per 100 | 18.5 | 15.0 | 21.1 | 33.4 | 34.5 | 28.1 |
| All errors per 100 | 40.7 | 28.6 | 50.3 | 71.4 | 73.9 | 66.3 |
| Error-free translations, % | 65.8 | 73.8 | 57.5 | 45.0 | 43.2 | 47.2 |
| Fluency, mean of 1 to 5 | 4.30 | 4.61 | 4.16 | 4.36 | 4.42 | 4.05 |
| Fluency 5, % | 41.5 | 66.3 | 32.2 | 50.0 | 49.8 | 25.2 |
| Fluency 4, % | 47.3 | 28.7 | 52.1 | 37.1 | 42.8 | 55.2 |
| Fluency 3, % | 10.7 | 4.5 | 15.1 | 11.8 | 7.1 | 19.0 |
| Fluency 2 or 1, % | 0.5 | 0.4 | 0.6 | 1.1 | 0.3 | 0.6 |
| Key items wrong per 100 | 5.9 | 4.4 | 3.0 | 10.8 | 13.3 | 13.5 |
| Translations with a key item wrong, % | 5.7 | 4.4 | 2.9 | 10.1 | 12.1 | 11.9 |
| Share of key items wrong, % | 1.19 | 0.89 | 0.61 | 2.16 | 2.67 | 2.71 |
| Unusable, count | 1 | 3 | 0 | 10 | 1 | 1 |
| Unusable, % | 0.10 | 0.30 | 0.00 | 1.01 | 0.10 | 0.10 |
| Composite, mean | 94.72 | 96.56 | 94.78 | 93.05 | 93.52 | 91.72 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 87.30 | 92.26 | 89.25 | 81.84 | 81.30 | 78.68 |
| Composite, double serious penalty | 93.78 | 95.79 | 94.36 | 91.43 | 91.43 | 89.66 |
| Serious fidelity errors per 100 | 3.5 | 1.9 | 1.6 | 7.7 | 8.4 | 7.5 |
| Serious logic errors per 100 | 0.7 | 0.2 | 0.2 | 0.7 | 0.9 | 1.5 |
| Serious numbers errors per 100 | 1.1 | 1.7 | 0.8 | 1.7 | 3.8 | 2.4 |
| Serious comprehension errors per 100 | 1.0 | 0.6 | 0.3 | 1.3 | 0.4 | 0.9 |
| All fidelity errors per 100 | 14.5 | 9.7 | 9.2 | 35.7 | 38.9 | 21.5 |
| All logic errors per 100 | 2.1 | 1.9 | 0.7 | 3.6 | 5.4 | 2.5 |
| All numbers errors per 100 | 3.6 | 3.5 | 2.7 | 4.6 | 9.1 | 7.9 |
| All comprehension errors per 100 | 18.4 | 11.3 | 23.1 | 22.1 | 17.9 | 32.0 |
| All form errors per 100 | 2.0 | 2.2 | 14.6 | 5.3 | 2.5 | 2.5 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -1.9 (-4.7 to +1.2) | -3.4 (-5.7 to -1.2) clear | +5.0 (+2.3 to +7.8) clear | +7.2 (+3.4 to +11.2) clear | +5.9 (+3.1 to +9.1) clear |
| Critical errors per 100 | +0.6 (-1.0 to +2.7) | -0.6 (-1.3 to +0.1) | +0.4 (-0.5 to +1.4) | +1.1 (-0.2 to +2.5) | +2.2 (+0.8 to +3.7) clear |
| Error-free, points | +8.0 (+3.9 to +12.0) clear | -8.4 (-14.3 to -2.3) clear | -20.9 (-25.9 to -16.2) clear | -22.6 (-27.1 to -17.8) clear | -18.6 (-23.5 to -14.2) clear |
| Fluency | +0.31 (+0.26 to +0.36) clear | -0.14 (-0.22 to -0.06) clear | +0.06 (-0.00 to +0.12) | +0.12 (+0.06 to +0.18) clear | -0.25 (-0.30 to -0.20) clear |
| Key items wrong per 100 | -1.5 (-4.0 to +1.3) | -2.9 (-5.1 to -0.6) clear | +4.8 (+2.3 to +7.5) clear | +7.4 (+4.1 to +10.9) clear | +7.6 (+4.7 to +10.8) clear |
| Composite | +1.83 (+0.83 to +2.71) clear | +0.06 (-0.71 to +0.88) | -1.67 (-2.66 to -0.69) clear | -1.21 (-2.22 to -0.23) clear | -3.00 (-3.93 to -2.12) clear |

### 5 experimental languages

750 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 7.2 | 3.9 | 6.7 | 15.1 | 15.7 | 16.1 |
| Translations with a serious error, % | 6.8 | 3.7 | 6.5 | 13.9 | 14.9 | 15.2 |
| Critical errors per 100 | 0.8 | 0.8 | 0.8 | 3.7 | 3.5 | 3.5 |
| Major errors per 100 | 6.4 | 3.1 | 5.9 | 11.3 | 12.3 | 12.7 |
| Minor errors per 100 | 11.5 | 8.5 | 9.9 | 22.9 | 11.6 | 23.1 |
| Cosmetic errors per 100 | 18.8 | 16.4 | 15.1 | 27.9 | 19.5 | 24.8 |
| All errors per 100 | 40.1 | 30.3 | 43.1 | 69.2 | 48.1 | 66.5 |
| Error-free translations, % | 64.7 | 73.6 | 61.3 | 47.1 | 60.0 | 47.7 |
| Fluency, mean of 1 to 5 | 4.33 | 4.57 | 4.23 | 4.21 | 4.47 | 4.04 |
| Fluency 5, % | 44.7 | 64.3 | 36.3 | 39.6 | 54.7 | 29.9 |
| Fluency 4, % | 44.7 | 29.1 | 51.5 | 43.7 | 38.5 | 47.2 |
| Fluency 3, % | 9.3 | 6.1 | 10.9 | 14.4 | 6.4 | 20.0 |
| Fluency 2 or 1, % | 1.3 | 0.5 | 1.3 | 2.3 | 0.4 | 2.9 |
| Key items wrong per 100 | 7.3 | 3.6 | 6.3 | 15.2 | 16.4 | 18.1 |
| Translations with a key item wrong, % | 7.1 | 3.6 | 6.0 | 13.7 | 15.2 | 15.7 |
| Share of key items wrong, % | 1.93 | 0.95 | 1.65 | 4.01 | 4.32 | 4.78 |
| Unusable, count | 0 | 0 | 0 | 4 | 1 | 2 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.53 | 0.13 | 0.27 |
| Composite, mean | 94.37 | 96.48 | 93.93 | 90.62 | 92.46 | 89.58 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 96.00 | 95.00 |
| Composite, lowest 10% cut-off | 86.67 | 91.00 | 87.06 | 71.54 | 73.33 | 70.00 |
| Composite, double serious penalty | 93.07 | 95.72 | 92.59 | 87.77 | 89.26 | 86.27 |
| Serious fidelity errors per 100 | 3.3 | 1.7 | 2.7 | 8.8 | 7.3 | 6.7 |
| Serious logic errors per 100 | 1.9 | 0.5 | 0.9 | 2.1 | 3.6 | 3.5 |
| Serious numbers errors per 100 | 1.6 | 1.3 | 2.5 | 2.5 | 4.4 | 4.5 |
| Serious comprehension errors per 100 | 0.4 | 0.3 | 0.5 | 1.6 | 0.4 | 1.5 |
| All fidelity errors per 100 | 11.5 | 11.2 | 7.6 | 28.8 | 24.4 | 20.5 |
| All logic errors per 100 | 3.7 | 1.6 | 2.4 | 5.1 | 5.3 | 5.5 |
| All numbers errors per 100 | 3.1 | 3.3 | 4.5 | 6.4 | 7.5 | 8.5 |
| All comprehension errors per 100 | 19.2 | 12.7 | 17.1 | 25.6 | 9.6 | 29.5 |
| All form errors per 100 | 2.7 | 1.5 | 11.5 | 3.3 | 1.3 | 2.5 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -3.3 (-6.0 to -0.4) clear | -0.5 (-3.6 to +2.8) | +7.9 (+4.0 to +12.0) clear | +8.5 (+4.9 to +12.8) clear | +8.9 (+5.3 to +12.9) clear |
| Critical errors per 100 | +0.0 (-1.1 to +1.2) | +0.0 (-1.1 to +1.1) | +2.9 (+0.9 to +5.2) clear | +2.7 (+0.8 to +5.1) clear | +2.7 (+0.9 to +4.8) clear |
| Error-free, points | +8.9 (+4.0 to +13.5) clear | -3.3 (-8.5 to +2.0) | -17.6 (-22.7 to -12.4) clear | -4.7 (-9.5 to +0.1) | -16.9 (-21.7 to -12.0) clear |
| Fluency | +0.24 (+0.18 to +0.31) clear | -0.10 (-0.17 to -0.03) clear | -0.12 (-0.19 to -0.05) clear | +0.15 (+0.07 to +0.22) clear | -0.29 (-0.36 to -0.21) clear |
| Key items wrong per 100 | -3.7 (-6.4 to -0.8) clear | -1.1 (-4.1 to +2.1) | +7.9 (+4.0 to +12.0) clear | +9.1 (+4.9 to +13.6) clear | +10.8 (+6.8 to +15.3) clear |
| Composite | +2.11 (+1.15 to +2.97) clear | -0.44 (-1.49 to +0.56) | -3.75 (-5.32 to -2.35) clear | -1.90 (-3.57 to -0.51) clear | -4.79 (-6.27 to -3.47) clear |

### The 8 languages tested first

1200 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 3.9 | 3.5 | 2.5 | 11.7 | 12.2 | 12.1 |
| Translations with a serious error, % | 3.8 | 3.5 | 2.3 | 10.6 | 11.5 | 11.0 |
| Critical errors per 100 | 0.8 | 1.3 | 0.2 | 2.7 | 2.0 | 3.3 |
| Major errors per 100 | 3.1 | 2.2 | 2.2 | 9.0 | 10.2 | 8.8 |
| Minor errors per 100 | 8.7 | 5.8 | 8.9 | 15.8 | 15.2 | 17.0 |
| Cosmetic errors per 100 | 12.2 | 10.6 | 13.9 | 24.6 | 23.7 | 19.9 |
| All errors per 100 | 26.5 | 21.1 | 38.8 | 55.4 | 52.6 | 50.9 |
| Error-free translations, % | 76.5 | 80.6 | 65.5 | 54.6 | 56.5 | 56.3 |
| Fluency, mean of 1 to 5 | 4.41 | 4.63 | 4.25 | 4.38 | 4.47 | 4.13 |
| Fluency 5, % | 49.3 | 67.6 | 38.8 | 52.3 | 53.5 | 31.8 |
| Fluency 4, % | 43.1 | 28.3 | 48.3 | 34.9 | 40.0 | 49.8 |
| Fluency 3, % | 7.1 | 3.9 | 12.3 | 10.8 | 6.2 | 17.6 |
| Fluency 2 or 1, % | 0.5 | 0.2 | 0.6 | 2.0 | 0.3 | 0.8 |
| Key items wrong per 100 | 4.0 | 3.7 | 2.4 | 11.6 | 12.3 | 12.3 |
| Translations with a key item wrong, % | 3.8 | 3.7 | 2.2 | 9.8 | 11.4 | 11.0 |
| Share of key items wrong, % | 1.05 | 0.97 | 0.64 | 3.05 | 3.25 | 3.25 |
| Unusable, count | 0 | 2 | 0 | 8 | 1 | 2 |
| Unusable, % | 0.00 | 0.17 | 0.00 | 0.67 | 0.08 | 0.17 |
| Composite, mean | 95.71 | 96.82 | 95.23 | 92.71 | 93.41 | 91.44 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 90.00 | 94.00 | 90.00 | 81.00 | 79.62 | 77.11 |
| Composite, double serious penalty | 94.94 | 96.05 | 94.75 | 90.61 | 91.00 | 89.01 |
| Serious fidelity errors per 100 | 1.8 | 1.7 | 1.4 | 6.4 | 7.6 | 6.2 |
| Serious logic errors per 100 | 0.8 | 0.5 | 0.2 | 0.8 | 1.8 | 2.4 |
| Serious numbers errors per 100 | 0.8 | 0.8 | 0.8 | 2.6 | 2.4 | 2.4 |
| Serious comprehension errors per 100 | 0.5 | 0.6 | 0.1 | 1.9 | 0.3 | 1.0 |
| All fidelity errors per 100 | 8.4 | 7.2 | 7.2 | 26.0 | 29.4 | 15.5 |
| All logic errors per 100 | 2.1 | 2.4 | 1.2 | 3.8 | 5.6 | 4.1 |
| All numbers errors per 100 | 2.3 | 1.7 | 3.3 | 4.7 | 5.3 | 6.2 |
| All comprehension errors per 100 | 12.0 | 8.5 | 13.6 | 17.6 | 10.7 | 23.2 |
| All form errors per 100 | 1.7 | 1.2 | 13.4 | 3.4 | 1.6 | 1.9 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -0.4 (-2.6 to +1.9) | -1.4 (-3.3 to +0.5) | +7.8 (+5.1 to +10.5) clear | +8.2 (+5.1 to +11.8) clear | +8.2 (+5.6 to +10.9) clear |
| Critical errors per 100 | +0.5 (-0.7 to +2.0) | -0.6 (-1.2 to +0.0) | +1.8 (+0.5 to +3.4) clear | +1.2 (-0.1 to +2.7) | +2.5 (+1.1 to +4.2) clear |
| Error-free, points | +4.1 (+0.7 to +7.4) clear | -11.0 (-15.7 to -6.5) clear | -21.9 (-25.6 to -18.6) clear | -20.0 (-23.8 to -16.4) clear | -20.2 (-24.0 to -16.2) clear |
| Fluency | +0.22 (+0.17 to +0.27) clear | -0.16 (-0.22 to -0.10) clear | -0.04 (-0.09 to +0.01) | +0.05 (-0.00 to +0.11) | -0.28 (-0.34 to -0.23) clear |
| Key items wrong per 100 | -0.3 (-2.6 to +2.2) | -1.6 (-3.5 to +0.3) | +7.6 (+4.5 to +11.0) clear | +8.3 (+5.1 to +12.1) clear | +8.3 (+5.8 to +11.2) clear |
| Composite | +1.10 (+0.26 to +1.82) clear | -0.48 (-1.13 to +0.19) | -3.00 (-4.22 to -1.97) clear | -2.30 (-3.47 to -1.29) clear | -4.27 (-5.33 to -3.31) clear |

### The 13 languages tested last

1950 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 5.7 | 3.5 | 5.1 | 11.4 | 14.1 | 13.6 |
| Translations with a serious error, % | 5.4 | 3.4 | 5.0 | 10.6 | 13.3 | 13.0 |
| Critical errors per 100 | 0.6 | 0.7 | 0.6 | 2.6 | 3.3 | 3.6 |
| Major errors per 100 | 5.1 | 2.8 | 4.5 | 8.8 | 10.8 | 10.0 |
| Minor errors per 100 | 10.2 | 6.3 | 8.9 | 17.0 | 14.3 | 20.0 |
| Cosmetic errors per 100 | 16.5 | 13.5 | 15.2 | 26.6 | 22.3 | 23.6 |
| All errors per 100 | 34.1 | 25.1 | 41.8 | 59.4 | 53.0 | 59.5 |
| Error-free translations, % | 70.4 | 77.1 | 62.9 | 52.6 | 56.8 | 52.3 |
| Fluency, mean of 1 to 5 | 4.37 | 4.61 | 4.26 | 4.31 | 4.46 | 4.10 |
| Fluency 5, % | 47.9 | 67.8 | 40.8 | 46.7 | 54.4 | 32.4 |
| Fluency 4, % | 42.2 | 26.2 | 45.2 | 39.0 | 37.6 | 47.4 |
| Fluency 3, % | 8.8 | 5.5 | 13.0 | 12.6 | 7.4 | 18.1 |
| Fluency 2 or 1, % | 1.1 | 0.5 | 1.0 | 1.7 | 0.6 | 2.1 |
| Key items wrong per 100 | 5.7 | 3.4 | 5.1 | 11.7 | 14.9 | 15.3 |
| Translations with a key item wrong, % | 5.4 | 3.3 | 5.0 | 10.5 | 13.7 | 13.5 |
| Share of key items wrong, % | 1.50 | 0.91 | 1.35 | 3.08 | 3.92 | 4.04 |
| Unusable, count | 1 | 1 | 0 | 19 | 3 | 6 |
| Unusable, % | 0.05 | 0.05 | 0.00 | 0.97 | 0.15 | 0.31 |
| Composite, mean | 95.02 | 96.83 | 94.54 | 91.90 | 92.73 | 90.64 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 87.47 | 92.06 | 87.37 | 77.59 | 75.00 | 72.30 |
| Composite, double serious penalty | 94.03 | 96.15 | 93.52 | 89.70 | 89.82 | 87.81 |
| Serious fidelity errors per 100 | 2.9 | 1.5 | 2.1 | 6.7 | 7.5 | 6.9 |
| Serious logic errors per 100 | 1.2 | 0.4 | 0.7 | 1.6 | 2.4 | 2.4 |
| Serious numbers errors per 100 | 0.9 | 1.4 | 1.9 | 2.1 | 3.4 | 3.3 |
| Serious comprehension errors per 100 | 0.6 | 0.2 | 0.4 | 1.0 | 0.7 | 1.0 |
| All fidelity errors per 100 | 11.4 | 8.5 | 6.6 | 25.3 | 25.3 | 19.3 |
| All logic errors per 100 | 2.9 | 1.2 | 1.5 | 4.1 | 5.3 | 3.8 |
| All numbers errors per 100 | 2.4 | 3.5 | 3.4 | 5.3 | 6.7 | 6.9 |
| All comprehension errors per 100 | 15.6 | 10.1 | 17.7 | 20.3 | 13.4 | 27.1 |
| All form errors per 100 | 1.7 | 1.8 | 12.6 | 4.5 | 2.3 | 2.3 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -2.2 (-4.1 to -0.1) clear | -0.6 (-2.3 to +1.2) | +5.7 (+3.3 to +8.4) clear | +8.4 (+5.7 to +11.6) clear | +7.9 (+5.5 to +10.6) clear |
| Critical errors per 100 | +0.1 (-0.7 to +1.2) | +0.1 (-0.6 to +0.7) | +2.1 (+0.6 to +3.8) clear | +2.7 (+1.2 to +4.6) clear | +3.0 (+1.6 to +4.8) clear |
| Error-free, points | +6.7 (+3.8 to +9.6) clear | -7.5 (-11.3 to -3.7) clear | -17.8 (-21.3 to -14.4) clear | -13.6 (-16.9 to -10.3) clear | -18.1 (-21.1 to -15.1) clear |
| Fluency | +0.24 (+0.21 to +0.28) clear | -0.11 (-0.16 to -0.06) clear | -0.06 (-0.11 to -0.02) clear | +0.09 (+0.04 to +0.13) clear | -0.27 (-0.31 to -0.22) clear |
| Key items wrong per 100 | -2.3 (-4.2 to -0.2) clear | -0.6 (-2.5 to +1.4) | +6.0 (+3.4 to +9.1) clear | +9.2 (+6.2 to +12.6) clear | +9.6 (+6.8 to +12.7) clear |
| Composite | +1.81 (+1.17 to +2.37) clear | -0.48 (-1.11 to +0.13) | -3.12 (-4.31 to -2.07) clear | -2.29 (-3.47 to -1.23) clear | -4.38 (-5.48 to -3.37) clear |

### Clean for OpenTalk (no fixes built from these)

2430 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 5.2 | 3.4 | 4.7 | 11.0 | 13.8 | 13.2 |
| Translations with a serious error, % | 4.9 | 3.3 | 4.5 | 10.2 | 13.0 | 12.6 |
| Critical errors per 100 | 0.7 | 0.8 | 0.5 | 2.5 | 2.9 | 3.4 |
| Major errors per 100 | 4.5 | 2.6 | 4.1 | 8.5 | 10.9 | 9.8 |
| Minor errors per 100 | 9.6 | 5.8 | 8.6 | 16.2 | 13.8 | 18.8 |
| Cosmetic errors per 100 | 15.5 | 12.5 | 14.7 | 26.3 | 22.3 | 22.4 |
| All errors per 100 | 31.9 | 23.4 | 40.9 | 57.6 | 51.9 | 56.6 |
| Error-free translations, % | 72.1 | 78.6 | 63.5 | 53.7 | 57.5 | 53.9 |
| Fluency, mean of 1 to 5 | 4.40 | 4.64 | 4.28 | 4.34 | 4.47 | 4.13 |
| Fluency 5, % | 49.5 | 69.5 | 41.8 | 49.2 | 55.4 | 33.6 |
| Fluency 4, % | 41.6 | 25.0 | 44.7 | 37.2 | 37.1 | 47.3 |
| Fluency 3, % | 7.9 | 5.1 | 12.7 | 12.0 | 7.0 | 17.3 |
| Fluency 2 or 1, % | 0.9 | 0.4 | 0.8 | 1.6 | 0.5 | 1.8 |
| Key items wrong per 100 | 5.2 | 3.4 | 4.7 | 11.2 | 14.3 | 14.7 |
| Translations with a key item wrong, % | 4.9 | 3.3 | 4.5 | 10.0 | 13.2 | 13.0 |
| Share of key items wrong, % | 1.38 | 0.90 | 1.24 | 2.98 | 3.81 | 3.92 |
| Unusable, count | 1 | 1 | 0 | 24 | 3 | 7 |
| Unusable, % | 0.04 | 0.04 | 0.00 | 0.99 | 0.12 | 0.29 |
| Composite, mean | 95.29 | 97.00 | 94.75 | 92.28 | 93.00 | 90.93 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 96.64 | 95.00 |
| Composite, lowest 10% cut-off | 89.00 | 94.00 | 88.00 | 79.33 | 76.07 | 73.33 |
| Composite, double serious penalty | 94.37 | 96.33 | 93.82 | 90.20 | 90.21 | 88.18 |
| Serious fidelity errors per 100 | 2.7 | 1.4 | 1.9 | 6.5 | 7.4 | 6.9 |
| Serious logic errors per 100 | 1.0 | 0.4 | 0.7 | 1.4 | 2.3 | 2.3 |
| Serious numbers errors per 100 | 0.9 | 1.3 | 1.8 | 2.0 | 3.5 | 3.1 |
| Serious comprehension errors per 100 | 0.6 | 0.3 | 0.4 | 1.2 | 0.6 | 0.9 |
| All fidelity errors per 100 | 11.0 | 8.0 | 6.6 | 25.3 | 25.3 | 18.6 |
| All logic errors per 100 | 2.5 | 1.2 | 1.4 | 3.7 | 5.0 | 3.7 |
| All numbers errors per 100 | 2.4 | 3.2 | 3.5 | 5.3 | 6.7 | 6.4 |
| All comprehension errors per 100 | 14.4 | 9.3 | 16.5 | 19.1 | 12.9 | 25.8 |
| All form errors per 100 | 1.6 | 1.7 | 12.9 | 4.2 | 2.0 | 2.1 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -1.8 (-3.4 to +0.1) | -0.5 (-2.1 to +1.2) | +5.8 (+3.8 to +8.1) clear | +8.6 (+5.9 to +11.7) clear | +8.0 (+5.7 to +10.6) clear |
| Critical errors per 100 | +0.1 (-0.7 to +1.1) | -0.1 (-0.7 to +0.4) | +1.8 (+0.6 to +3.3) clear | +2.3 (+1.0 to +3.9) clear | +2.8 (+1.6 to +4.2) clear |
| Error-free, points | +6.5 (+3.9 to +8.9) clear | -8.6 (-12.6 to -4.6) clear | -18.4 (-21.6 to -15.3) clear | -14.6 (-17.7 to -11.5) clear | -18.3 (-21.3 to -15.2) clear |
| Fluency | +0.24 (+0.21 to +0.28) clear | -0.12 (-0.18 to -0.07) clear | -0.06 (-0.10 to -0.02) clear | +0.08 (+0.04 to +0.12) clear | -0.27 (-0.31 to -0.22) clear |
| Key items wrong per 100 | -1.8 (-3.4 to +0.0) | -0.5 (-2.2 to +1.3) | +6.0 (+3.7 to +8.7) clear | +9.1 (+6.4 to +12.3) clear | +9.5 (+6.9 to +12.5) clear |
| Composite | +1.71 (+1.12 to +2.23) clear | -0.54 (-1.12 to +0.05) | -3.01 (-4.05 to -2.10) clear | -2.29 (-3.37 to -1.36) clear | -4.36 (-5.40 to -3.45) clear |

### Everyday texts (f001-f080)

1680 translations per system, from 80 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 5.0 | 3.2 | 3.4 | 12.3 | 11.5 | 12.1 |
| Translations with a serious error, % | 4.8 | 3.1 | 3.3 | 11.1 | 10.7 | 11.3 |
| Critical errors per 100 | 0.5 | 1.0 | 0.4 | 3.0 | 1.8 | 3.9 |
| Major errors per 100 | 4.5 | 2.1 | 3.0 | 9.2 | 9.7 | 8.3 |
| Minor errors per 100 | 10.6 | 6.4 | 9.2 | 17.7 | 16.3 | 20.7 |
| Cosmetic errors per 100 | 15.3 | 13.3 | 15.5 | 26.2 | 24.5 | 24.3 |
| All errors per 100 | 32.8 | 24.8 | 40.4 | 60.8 | 54.2 | 59.4 |
| Error-free translations, % | 71.6 | 77.6 | 64.5 | 51.4 | 55.7 | 51.1 |
| Fluency, mean of 1 to 5 | 4.36 | 4.60 | 4.22 | 4.31 | 4.44 | 4.05 |
| Fluency 5, % | 45.9 | 65.2 | 37.2 | 47.1 | 52.5 | 28.2 |
| Fluency 4, % | 44.8 | 29.7 | 48.9 | 38.7 | 39.3 | 50.3 |
| Fluency 3, % | 8.5 | 4.7 | 12.8 | 12.3 | 7.6 | 19.9 |
| Fluency 2 or 1, % | 0.9 | 0.4 | 1.1 | 2.0 | 0.6 | 1.6 |
| Key items wrong per 100 | 5.0 | 3.2 | 3.4 | 12.5 | 12.3 | 13.4 |
| Translations with a key item wrong, % | 4.8 | 3.1 | 3.3 | 10.8 | 11.2 | 11.8 |
| Share of key items wrong, % | 1.27 | 0.80 | 0.86 | 3.18 | 3.14 | 3.41 |
| Unusable, count | 1 | 2 | 0 | 14 | 2 | 2 |
| Unusable, % | 0.06 | 0.12 | 0.00 | 0.83 | 0.12 | 0.12 |
| Composite, mean | 95.19 | 96.80 | 94.85 | 91.95 | 93.47 | 90.87 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 88.67 | 92.44 | 89.00 | 78.01 | 81.84 | 77.21 |
| Composite, double serious penalty | 94.37 | 96.17 | 94.21 | 89.78 | 91.32 | 88.33 |
| Serious fidelity errors per 100 | 3.1 | 1.5 | 1.8 | 7.3 | 7.4 | 6.3 |
| Serious logic errors per 100 | 0.5 | 0.4 | 0.3 | 1.0 | 1.1 | 1.9 |
| Serious numbers errors per 100 | 0.7 | 1.0 | 1.0 | 2.7 | 2.2 | 2.8 |
| Serious comprehension errors per 100 | 0.7 | 0.2 | 0.3 | 1.2 | 0.8 | 1.1 |
| All fidelity errors per 100 | 11.4 | 8.5 | 7.0 | 28.2 | 29.8 | 19.2 |
| All logic errors per 100 | 1.7 | 1.6 | 1.0 | 3.2 | 4.0 | 2.9 |
| All numbers errors per 100 | 2.4 | 2.2 | 3.0 | 5.1 | 4.9 | 6.7 |
| All comprehension errors per 100 | 15.4 | 10.6 | 17.0 | 19.8 | 13.6 | 28.3 |
| All form errors per 100 | 1.9 | 1.8 | 12.3 | 4.7 | 1.9 | 2.3 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -1.8 (-3.9 to +0.6) | -1.6 (-3.5 to +0.4) | +7.3 (+4.0 to +11.1) clear | +6.5 (+3.5 to +10.0) clear | +7.1 (+4.1 to +10.6) clear |
| Critical errors per 100 | +0.5 (-0.6 to +2.0) | -0.1 (-0.7 to +0.4) | +2.5 (+0.5 to +5.6) clear | +1.3 (+0.2 to +2.6) clear | +3.3 (+1.2 to +6.3) clear |
| Error-free, points | +6.0 (+3.1 to +9.1) clear | -7.1 (-11.9 to -2.4) clear | -20.2 (-23.9 to -16.3) clear | -16.0 (-19.8 to -11.9) clear | -20.5 (-24.4 to -17.1) clear |
| Fluency | +0.24 (+0.19 to +0.28) clear | -0.13 (-0.19 to -0.07) clear | -0.05 (-0.10 to +0.01) | +0.08 (+0.02 to +0.15) clear | -0.31 (-0.36 to -0.25) clear |
| Key items wrong per 100 | -1.8 (-3.9 to +0.4) | -1.6 (-3.6 to +0.5) | +7.5 (+3.8 to +12.8) clear | +7.3 (+4.0 to +11.0) clear | +8.4 (+5.4 to +11.8) clear |
| Composite | +1.61 (+0.87 to +2.31) clear | -0.34 (-1.02 to +0.38) | -3.23 (-5.07 to -1.82) clear | -1.71 (-2.92 to -0.65) clear | -4.32 (-5.85 to -3.04) clear |

### Small-word texts (g001-g060)

1260 translations per system, from 60 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 3.9 | 3.2 | 4.4 | 8.2 | 13.3 | 12.5 |
| Translations with a serious error, % | 3.7 | 3.2 | 4.4 | 7.9 | 12.8 | 12.0 |
| Critical errors per 100 | 0.6 | 0.6 | 0.5 | 1.2 | 2.2 | 2.5 |
| Major errors per 100 | 3.3 | 2.6 | 4.0 | 7.0 | 11.0 | 10.0 |
| Minor errors per 100 | 7.3 | 5.1 | 7.1 | 14.1 | 11.3 | 15.6 |
| Cosmetic errors per 100 | 13.6 | 11.3 | 14.2 | 24.8 | 20.9 | 19.8 |
| All errors per 100 | 26.3 | 21.0 | 38.2 | 50.4 | 47.4 | 49.5 |
| Error-free translations, % | 76.4 | 80.1 | 64.8 | 57.9 | 60.2 | 58.8 |
| Fluency, mean of 1 to 5 | 4.45 | 4.67 | 4.33 | 4.39 | 4.51 | 4.21 |
| Fluency 5, % | 53.3 | 72.3 | 45.5 | 52.9 | 57.4 | 38.7 |
| Fluency 4, % | 38.8 | 22.7 | 42.8 | 34.6 | 36.2 | 44.9 |
| Fluency 3, % | 7.2 | 4.7 | 11.3 | 11.0 | 6.1 | 15.1 |
| Fluency 2 or 1, % | 0.7 | 0.3 | 0.4 | 1.5 | 0.3 | 1.3 |
| Key items wrong per 100 | 3.8 | 3.3 | 4.3 | 8.4 | 13.2 | 13.4 |
| Translations with a key item wrong, % | 3.7 | 3.2 | 4.1 | 7.5 | 12.4 | 12.0 |
| Share of key items wrong, % | 1.05 | 0.90 | 1.18 | 2.33 | 3.64 | 3.71 |
| Unusable, count | 0 | 1 | 0 | 13 | 1 | 4 |
| Unusable, % | 0.00 | 0.08 | 0.00 | 1.03 | 0.08 | 0.32 |
| Composite, mean | 95.96 | 97.23 | 95.16 | 93.44 | 93.59 | 91.84 |
| Composite, median | 99.30 | 100.00 | 95.00 | 96.67 | 99.00 | 95.00 |
| Composite, lowest 10% cut-off | 90.00 | 94.00 | 89.00 | 85.89 | 77.11 | 75.29 |
| Composite, double serious penalty | 95.24 | 96.63 | 94.26 | 91.95 | 91.03 | 89.41 |
| Serious fidelity errors per 100 | 1.7 | 1.0 | 1.5 | 4.6 | 7.1 | 6.3 |
| Serious logic errors per 100 | 0.6 | 0.2 | 0.5 | 0.8 | 1.5 | 2.1 |
| Serious numbers errors per 100 | 1.1 | 1.5 | 2.2 | 1.5 | 4.4 | 3.4 |
| Serious comprehension errors per 100 | 0.6 | 0.5 | 0.2 | 1.3 | 0.2 | 0.6 |
| All fidelity errors per 100 | 9.1 | 6.7 | 6.1 | 21.0 | 22.0 | 15.5 |
| All logic errors per 100 | 1.6 | 1.0 | 1.0 | 3.0 | 3.9 | 3.3 |
| All numbers errors per 100 | 2.5 | 3.9 | 4.4 | 5.2 | 8.4 | 7.0 |
| All comprehension errors per 100 | 11.6 | 8.0 | 14.3 | 17.9 | 11.2 | 22.0 |
| All form errors per 100 | 1.6 | 1.5 | 12.4 | 3.3 | 1.9 | 1.7 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -0.7 (-2.5 to +1.3) | +0.6 (-1.7 to +2.9) | +4.3 (+2.1 to +6.7) clear | +9.4 (+5.4 to +14.2) clear | +8.6 (+5.6 to +12.1) clear |
| Critical errors per 100 | -0.1 (-1.0 to +1.0) | -0.2 (-1.0 to +0.6) | +0.6 (-0.2 to +1.3) | +1.6 (+0.4 to +2.9) clear | +1.8 (+0.7 to +3.0) clear |
| Error-free, points | +3.7 (+0.3 to +7.1) clear | -11.6 (-17.8 to -6.0) clear | -18.5 (-22.9 to -14.2) clear | -16.2 (-20.9 to -11.7) clear | -17.6 (-22.9 to -13.0) clear |
| Fluency | +0.22 (+0.17 to +0.27) clear | -0.11 (-0.19 to -0.04) clear | -0.06 (-0.12 to +0.00) | +0.06 (+0.01 to +0.12) clear | -0.24 (-0.30 to -0.17) clear |
| Key items wrong per 100 | -0.6 (-2.2 to +1.5) | +0.5 (-1.7 to +2.8) | +4.6 (+2.2 to +7.1) clear | +9.4 (+5.6 to +14.0) clear | +9.6 (+6.2 to +13.6) clear |
| Composite | +1.27 (+0.55 to +1.91) clear | -0.80 (-1.62 to -0.03) clear | -2.52 (-3.59 to -1.63) clear | -2.37 (-3.74 to -1.20) clear | -4.12 (-5.49 to -2.97) clear |

### Difficult passages (f081-f090)

210 translations per system, from 10 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 11.9 | 8.1 | 7.6 | 25.2 | 28.6 | 23.3 |
| Translations with a serious error, % | 11.9 | 8.1 | 7.1 | 22.9 | 27.1 | 21.0 |
| Critical errors per 100 | 1.9 | 2.4 | 1.0 | 8.1 | 13.8 | 6.7 |
| Major errors per 100 | 10.0 | 5.7 | 6.7 | 17.1 | 14.8 | 16.7 |
| Minor errors per 100 | 15.2 | 9.0 | 17.6 | 21.4 | 21.0 | 24.3 |
| Cosmetic errors per 100 | 19.0 | 11.9 | 11.9 | 29.0 | 21.4 | 20.0 |
| All errors per 100 | 47.1 | 29.5 | 57.6 | 79.0 | 74.8 | 71.0 |
| Error-free translations, % | 59.5 | 75.2 | 53.8 | 41.0 | 43.3 | 46.2 |
| Fluency, mean of 1 to 5 | 4.26 | 4.52 | 4.06 | 4.19 | 4.38 | 4.00 |
| Fluency 5, % | 40.0 | 60.5 | 30.0 | 38.6 | 46.2 | 25.2 |
| Fluency 4, % | 47.1 | 31.4 | 47.6 | 44.3 | 46.2 | 52.9 |
| Fluency 3, % | 11.4 | 8.1 | 21.0 | 14.8 | 7.1 | 19.0 |
| Fluency 2 or 1, % | 1.4 | 0.0 | 1.4 | 2.4 | 0.5 | 2.9 |
| Key items wrong per 100 | 12.9 | 8.1 | 8.6 | 24.3 | 31.0 | 25.2 |
| Translations with a key item wrong, % | 12.4 | 8.1 | 8.1 | 21.9 | 28.1 | 21.4 |
| Share of key items wrong, % | 3.38 | 2.13 | 2.26 | 6.39 | 8.15 | 6.64 |
| Unusable, count | 0 | 0 | 0 | 0 | 1 | 2 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.48 | 0.95 |
| Composite, mean | 92.07 | 94.62 | 92.33 | 86.88 | 85.60 | 86.27 |
| Composite, median | 95.00 | 100.00 | 95.00 | 94.00 | 95.00 | 94.00 |
| Composite, lowest 10% cut-off | 74.00 | 86.00 | 86.00 | 61.00 | 54.00 | 61.00 |
| Composite, double serious penalty | 89.30 | 92.53 | 90.62 | 80.78 | 77.35 | 80.93 |
| Serious fidelity errors per 100 | 2.9 | 4.8 | 3.8 | 12.9 | 11.4 | 11.9 |
| Serious logic errors per 100 | 8.1 | 2.9 | 2.9 | 6.7 | 15.2 | 7.6 |
| Serious numbers errors per 100 | 1.0 | 0.0 | 0.5 | 3.3 | 1.4 | 1.9 |
| Serious comprehension errors per 100 | 0.0 | 0.5 | 0.5 | 2.4 | 0.5 | 1.9 |
| All fidelity errors per 100 | 8.1 | 12.4 | 9.5 | 32.4 | 32.9 | 21.9 |
| All logic errors per 100 | 15.2 | 6.7 | 7.1 | 15.7 | 25.2 | 15.2 |
| All numbers errors per 100 | 1.9 | 1.0 | 0.5 | 4.3 | 2.9 | 3.8 |
| All comprehension errors per 100 | 21.0 | 9.0 | 20.0 | 23.3 | 10.0 | 26.7 |
| All form errors per 100 | 1.0 | 0.5 | 20.5 | 3.3 | 3.8 | 3.3 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -3.8 (-13.8 to +5.7) | -4.3 (-13.8 to +2.4) | +13.3 (+1.9 to +27.6) clear | +16.7 (-1.0 to +38.6) | +11.4 (-1.4 to +25.2) |
| Critical errors per 100 | +0.5 (-1.9 to +3.8) | -1.0 (-2.4 to +0.0) | +6.2 (+1.0 to +13.8) clear | +11.9 (-2.4 to +31.4) | +4.8 (+0.0 to +10.5) |
| Error-free, points | +15.7 (+5.2 to +26.2) clear | -5.7 (-16.2 to +4.8) | -18.6 (-34.8 to -5.7) clear | -16.2 (-33.3 to -1.4) clear | -13.3 (-25.2 to -1.9) clear |
| Fluency | +0.27 (+0.11 to +0.43) clear | -0.20 (-0.41 to +0.04) | -0.07 (-0.25 to +0.12) | +0.12 (-0.02 to +0.28) | -0.25 (-0.39 to -0.14) clear |
| Key items wrong per 100 | -4.8 (-17.1 to +6.2) | -4.3 (-16.2 to +4.8) | +11.4 (-2.4 to +26.2) | +18.1 (-1.9 to +42.4) | +12.4 (-3.8 to +29.5) |
| Composite | +2.56 (-0.79 to +5.79) | +0.27 (-2.01 to +3.01) | -5.19 (-10.71 to -1.11) clear | -6.46 (-16.77 to +1.68) | -5.79 (-10.69 to -1.67) clear |

## Serious errors per 100, by language and system

### Serious errors per 100

| Language | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Italian | 1.3 | 4.0 | 1.3 | 7.3 | 10.7 | 6.7 |
| Vietnamese | 1.3 | 1.3 | 1.3 | 8.7 | 10.0 | 14.0 |
| French | 2.0 | 4.0 | 3.3 | 6.0 | 10.0 | 6.0 |
| Latin American Spanish | 2.0 | 0.0 | 1.3 | 4.7 | 8.0 | 2.0 |
| Simplified Chinese | 2.7 | 2.7 | 4.7 | 6.7 | 13.3 | 24.0 |
| Brazilian Portuguese | 3.3 | 4.0 | 4.0 | 6.0 | 11.3 | 9.3 |
| European Spanish | 4.0 | 3.3 | 2.7 | 6.7 | 9.3 | 4.7 |
| Traditional Chinese | 4.0 | 3.3 | 6.0 | 7.3 | 16.7 | 22.0 |
| Hindi | 4.7 | 2.7 | 1.3 | 28.7 | 15.3 | 12.7 |
| Arabic | 5.3 | 4.7 | 1.3 | 15.3 | 11.3 | 9.3 |
| Indonesian | 5.3 | 2.0 | 4.7 | 7.3 | 10.7 | 6.7 |
| Thai | 6.0 | 4.0 | 2.0 | 17.3 | 14.7 | 14.0 |
| Turkish | 6.0 | 4.7 | 4.0 | 10.7 | 15.3 | 17.3 |
| Ukrainian | 6.0 | 3.3 | 5.3 | 13.3 | 17.3 | 10.7 |
| German | 6.7 | 4.0 | 5.3 | 10.7 | 14.7 | 15.3 |
| Persian | 6.7 | 6.7 | 3.3 | 18.0 | 14.0 | 17.3 |
| Russian | 6.7 | 6.0 | 3.3 | 5.3 | 15.3 | 11.3 |
| Dutch | 7.3 | 2.0 | 6.0 | 14.0 | 12.0 | 16.7 |
| Polish | 7.3 | 1.3 | 6.7 | 10.0 | 16.7 | 17.3 |
| Korean | 8.0 | 4.0 | 9.3 | 22.0 | 18.0 | 12.7 |
| Japanese | 8.7 | 5.3 | 8.7 | 15.3 | 16.0 | 23.3 |

### Critical errors per 100

| Language | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Italian | 0.0 | 0.7 | 0.0 | 1.3 | 2.7 | 2.0 |
| Vietnamese | 0.0 | 1.3 | 0.0 | 2.7 | 0.7 | 2.7 |
| French | 0.7 | 2.0 | 0.0 | 0.7 | 1.3 | 2.7 |
| Latin American Spanish | 0.7 | 0.0 | 0.0 | 0.7 | 1.3 | 1.3 |
| Simplified Chinese | 0.0 | 0.7 | 0.7 | 2.7 | 2.7 | 11.3 |
| Brazilian Portuguese | 0.0 | 0.7 | 1.3 | 0.7 | 6.0 | 2.7 |
| European Spanish | 0.7 | 0.7 | 0.7 | 0.7 | 1.3 | 2.0 |
| Traditional Chinese | 0.7 | 0.7 | 0.0 | 0.7 | 2.7 | 8.7 |
| Hindi | 0.7 | 0.7 | 0.7 | 4.7 | 2.0 | 2.7 |
| Arabic | 2.0 | 0.7 | 0.0 | 2.0 | 2.7 | 1.3 |
| Indonesian | 0.0 | 0.0 | 0.7 | 2.7 | 1.3 | 1.3 |
| Thai | 0.7 | 0.7 | 0.0 | 4.7 | 4.7 | 3.3 |
| Turkish | 0.0 | 0.0 | 0.0 | 2.7 | 3.3 | 5.3 |
| Ukrainian | 0.0 | 0.7 | 1.3 | 4.0 | 2.0 | 2.7 |
| German | 0.0 | 0.7 | 1.3 | 3.3 | 3.3 | 5.3 |
| Persian | 2.0 | 3.3 | 0.0 | 8.0 | 3.3 | 2.7 |
| Russian | 0.7 | 2.0 | 0.7 | 0.0 | 2.0 | 2.0 |
| Dutch | 0.0 | 0.7 | 0.7 | 2.0 | 2.0 | 2.7 |
| Polish | 1.3 | 0.7 | 0.0 | 1.3 | 3.3 | 4.0 |
| Korean | 1.3 | 1.3 | 0.7 | 5.3 | 4.0 | 3.3 |
| Japanese | 2.7 | 1.3 | 1.3 | 4.7 | 6.0 | 3.3 |

### Error-free, %

| Language | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Italian | 78.0 | 72.0 | 66.0 | 56.7 | 46.7 | 46.7 |
| Vietnamese | 87.3 | 84.7 | 67.3 | 64.7 | 59.3 | 59.3 |
| French | 75.3 | 81.3 | 69.3 | 61.3 | 58.0 | 62.7 |
| Latin American Spanish | 85.3 | 88.7 | 63.3 | 64.0 | 66.0 | 65.3 |
| Simplified Chinese | 78.7 | 88.7 | 58.7 | 68.0 | 59.3 | 48.0 |
| Brazilian Portuguese | 86.7 | 84.0 | 70.7 | 66.0 | 68.0 | 66.7 |
| European Spanish | 80.7 | 85.3 | 66.0 | 64.0 | 67.3 | 71.3 |
| Traditional Chinese | 81.3 | 86.0 | 54.0 | 50.7 | 52.7 | 50.0 |
| Hindi | 74.7 | 81.3 | 55.3 | 30.7 | 50.0 | 62.0 |
| Arabic | 72.0 | 77.3 | 74.0 | 42.0 | 50.0 | 54.7 |
| Indonesian | 78.0 | 83.3 | 62.0 | 71.3 | 61.3 | 68.0 |
| Thai | 71.3 | 81.3 | 84.0 | 44.0 | 52.0 | 58.0 |
| Turkish | 63.3 | 68.0 | 61.3 | 60.7 | 64.0 | 46.7 |
| Ukrainian | 64.7 | 65.3 | 58.7 | 34.7 | 56.0 | 44.0 |
| German | 68.0 | 79.3 | 61.3 | 53.3 | 51.3 | 44.7 |
| Persian | 68.0 | 71.3 | 65.3 | 50.0 | 54.7 | 52.0 |
| Russian | 70.7 | 71.3 | 70.7 | 56.0 | 54.7 | 46.7 |
| Dutch | 60.7 | 70.0 | 61.3 | 49.3 | 59.3 | 47.3 |
| Polish | 48.0 | 63.3 | 47.3 | 42.0 | 38.7 | 36.0 |
| Korean | 65.3 | 82.7 | 61.3 | 44.0 | 58.0 | 55.3 |
| Japanese | 69.3 | 82.0 | 64.0 | 46.7 | 62.7 | 45.3 |

### Fluency

| Language | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Italian | 4.43 | 4.59 | 4.43 | 4.39 | 4.31 | 3.98 |
| Vietnamese | 4.57 | 4.54 | 4.23 | 4.45 | 4.47 | 4.16 |
| French | 4.41 | 4.78 | 4.49 | 4.69 | 4.61 | 4.28 |
| Latin American Spanish | 4.61 | 4.75 | 4.46 | 4.57 | 4.65 | 4.42 |
| Simplified Chinese | 4.45 | 4.71 | 4.06 | 4.69 | 4.66 | 4.13 |
| Brazilian Portuguese | 4.66 | 4.80 | 4.49 | 4.62 | 4.70 | 4.30 |
| European Spanish | 4.63 | 4.76 | 4.51 | 4.59 | 4.69 | 4.43 |
| Traditional Chinese | 4.40 | 4.79 | 3.90 | 4.19 | 4.41 | 4.07 |
| Hindi | 4.44 | 4.61 | 3.95 | 3.62 | 4.05 | 3.97 |
| Arabic | 4.41 | 4.61 | 4.37 | 4.21 | 4.41 | 4.17 |
| Indonesian | 4.51 | 4.69 | 4.25 | 4.62 | 4.61 | 4.37 |
| Thai | 4.25 | 4.52 | 4.43 | 4.19 | 4.30 | 4.25 |
| Turkish | 4.22 | 4.51 | 4.25 | 4.43 | 4.53 | 4.03 |
| Ukrainian | 4.23 | 4.37 | 4.20 | 3.97 | 4.47 | 3.83 |
| German | 4.29 | 4.66 | 4.29 | 4.35 | 4.47 | 3.89 |
| Persian | 4.09 | 4.44 | 4.19 | 4.25 | 4.39 | 3.95 |
| Russian | 4.30 | 4.63 | 4.27 | 4.51 | 4.49 | 3.95 |
| Dutch | 4.27 | 4.53 | 4.27 | 4.15 | 4.43 | 3.98 |
| Polish | 4.00 | 4.31 | 3.95 | 4.01 | 4.09 | 3.82 |
| Korean | 4.46 | 4.71 | 4.19 | 4.09 | 4.43 | 4.35 |
| Japanese | 4.45 | 4.73 | 4.23 | 4.39 | 4.51 | 4.01 |

### Key items wrong per 100

| Language | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Italian | 1.3 | 4.0 | 3.3 | 7.3 | 10.7 | 9.3 |
| Vietnamese | 2.0 | 1.3 | 0.7 | 8.0 | 11.3 | 13.3 |
| French | 2.0 | 4.0 | 3.3 | 6.0 | 8.7 | 6.7 |
| Latin American Spanish | 2.0 | 0.0 | 1.3 | 6.0 | 7.3 | 2.7 |
| Simplified Chinese | 2.7 | 2.7 | 4.7 | 7.3 | 12.0 | 22.0 |
| Brazilian Portuguese | 4.0 | 4.0 | 4.0 | 6.0 | 12.7 | 10.0 |
| European Spanish | 3.3 | 3.3 | 3.3 | 6.7 | 9.3 | 5.3 |
| Traditional Chinese | 4.7 | 4.0 | 6.7 | 6.7 | 16.0 | 24.0 |
| Hindi | 4.7 | 3.3 | 1.3 | 27.3 | 17.3 | 14.7 |
| Arabic | 6.0 | 4.0 | 1.3 | 14.0 | 10.7 | 10.0 |
| Indonesian | 5.3 | 1.3 | 4.7 | 6.7 | 11.3 | 8.7 |
| Thai | 7.3 | 3.3 | 2.0 | 19.3 | 16.0 | 18.0 |
| Turkish | 6.0 | 4.7 | 4.7 | 10.0 | 18.0 | 23.3 |
| Ukrainian | 6.0 | 2.7 | 4.7 | 10.7 | 17.3 | 12.0 |
| German | 6.0 | 5.3 | 5.3 | 12.7 | 15.3 | 17.3 |
| Persian | 6.0 | 8.0 | 3.3 | 17.3 | 15.3 | 18.0 |
| Russian | 6.7 | 6.0 | 3.3 | 6.7 | 16.0 | 11.3 |
| Dutch | 6.0 | 2.7 | 5.3 | 15.3 | 12.7 | 17.3 |
| Polish | 5.3 | 1.3 | 6.0 | 10.7 | 20.0 | 16.0 |
| Korean | 8.7 | 3.3 | 8.7 | 23.3 | 20.7 | 14.7 |
| Japanese | 10.0 | 4.7 | 8.0 | 16.7 | 13.3 | 23.3 |

### Unusable, count

| Language | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Italian | 0 | 0 | 0 | 0 | 0 | 0 |
| Vietnamese | 0 | 0 | 0 | 2 | 0 | 0 |
| French | 0 | 0 | 0 | 0 | 0 | 0 |
| Latin American Spanish | 0 | 1 | 0 | 0 | 0 | 0 |
| Simplified Chinese | 0 | 0 | 0 | 0 | 1 | 0 |
| Brazilian Portuguese | 0 | 0 | 0 | 0 | 1 | 1 |
| European Spanish | 0 | 0 | 0 | 0 | 0 | 0 |
| Traditional Chinese | 0 | 1 | 0 | 11 | 0 | 2 |
| Hindi | 0 | 0 | 0 | 3 | 0 | 1 |
| Arabic | 0 | 0 | 0 | 1 | 0 | 0 |
| Indonesian | 0 | 0 | 0 | 1 | 0 | 0 |
| Thai | 1 | 0 | 0 | 3 | 1 | 0 |
| Turkish | 0 | 0 | 0 | 0 | 0 | 0 |
| Ukrainian | 0 | 0 | 0 | 1 | 0 | 2 |
| German | 0 | 0 | 0 | 0 | 0 | 1 |
| Persian | 0 | 1 | 0 | 1 | 0 | 0 |
| Russian | 0 | 0 | 0 | 1 | 0 | 1 |
| Dutch | 0 | 0 | 0 | 0 | 0 | 0 |
| Polish | 0 | 0 | 0 | 0 | 0 | 0 |
| Korean | 0 | 0 | 0 | 2 | 1 | 0 |
| Japanese | 0 | 0 | 0 | 1 | 0 | 0 |

### Composite

| Language | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Italian | 96.51 | 96.60 | 96.24 | 94.32 | 92.97 | 91.84 |
| Vietnamese | 97.37 | 96.98 | 95.50 | 93.51 | 94.13 | 91.37 |
| French | 96.06 | 97.55 | 96.42 | 96.40 | 94.83 | 93.81 |
| Latin American Spanish | 97.44 | 97.97 | 96.59 | 96.10 | 95.69 | 95.81 |
| Simplified Chinese | 96.36 | 97.67 | 93.56 | 95.81 | 93.92 | 87.32 |
| Brazilian Portuguese | 97.38 | 97.81 | 95.96 | 96.11 | 94.13 | 93.30 |
| European Spanish | 97.00 | 97.93 | 96.47 | 96.02 | 95.65 | 95.31 |
| Traditional Chinese | 95.80 | 97.32 | 92.50 | 87.29 | 92.21 | 87.06 |
| Hindi | 95.73 | 97.07 | 93.90 | 83.99 | 90.32 | 90.30 |
| Arabic | 95.08 | 96.64 | 96.38 | 91.04 | 93.14 | 92.77 |
| Indonesian | 96.12 | 97.83 | 94.78 | 95.47 | 94.73 | 94.29 |
| Thai | 93.91 | 96.51 | 96.38 | 88.20 | 91.02 | 91.59 |
| Turkish | 94.16 | 95.99 | 94.84 | 93.91 | 92.87 | 89.21 |
| Ukrainian | 94.28 | 95.49 | 93.99 | 89.37 | 92.48 | 89.06 |
| German | 94.50 | 96.75 | 94.62 | 92.82 | 92.64 | 88.58 |
| Persian | 93.10 | 94.47 | 94.61 | 89.85 | 92.39 | 89.76 |
| Russian | 94.56 | 96.18 | 94.91 | 94.97 | 92.88 | 90.43 |
| Dutch | 94.25 | 96.57 | 94.40 | 91.01 | 93.46 | 89.44 |
| Polish | 92.25 | 95.64 | 92.40 | 91.39 | 89.89 | 88.53 |
| Korean | 94.70 | 97.23 | 93.05 | 87.75 | 91.14 | 92.17 |
| Japanese | 94.45 | 97.13 | 93.37 | 91.06 | 92.37 | 88.02 |

### Serious errors per 100 up to 40 words

| Language | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Italian | 1.1 | 4.5 | 1.1 | 6.8 | 6.8 | 5.7 |
| Vietnamese | 0.0 | 1.1 | 2.3 | 9.1 | 11.4 | 14.8 |
| French | 2.3 | 2.3 | 2.3 | 4.5 | 10.2 | 8.0 |
| Latin American Spanish | 0.0 | 0.0 | 2.3 | 5.7 | 8.0 | 3.4 |
| Simplified Chinese | 2.3 | 2.3 | 3.4 | 8.0 | 10.2 | 27.3 |
| Brazilian Portuguese | 2.3 | 3.4 | 5.7 | 6.8 | 13.6 | 10.2 |
| European Spanish | 2.3 | 1.1 | 2.3 | 5.7 | 9.1 | 3.4 |
| Traditional Chinese | 2.3 | 1.1 | 4.5 | 6.8 | 13.6 | 25.0 |
| Hindi | 1.1 | 2.3 | 1.1 | 25.0 | 15.9 | 11.4 |
| Arabic | 4.5 | 2.3 | 0.0 | 8.0 | 12.5 | 6.8 |
| Indonesian | 3.4 | 2.3 | 4.5 | 8.0 | 11.4 | 10.2 |
| Thai | 4.5 | 3.4 | 3.4 | 20.5 | 13.6 | 13.6 |
| Turkish | 5.7 | 4.5 | 4.5 | 5.7 | 13.6 | 13.6 |
| Ukrainian | 4.5 | 1.1 | 5.7 | 14.8 | 17.0 | 10.2 |
| German | 4.5 | 3.4 | 5.7 | 10.2 | 14.8 | 17.0 |
| Persian | 6.8 | 4.5 | 4.5 | 15.9 | 13.6 | 13.6 |
| Russian | 3.4 | 6.8 | 5.7 | 4.5 | 13.6 | 9.1 |
| Dutch | 6.8 | 1.1 | 6.8 | 15.9 | 11.4 | 18.2 |
| Polish | 5.7 | 1.1 | 8.0 | 9.1 | 13.6 | 10.2 |
| Korean | 9.1 | 3.4 | 11.4 | 26.1 | 20.5 | 17.0 |
| Japanese | 5.7 | 3.4 | 11.4 | 18.2 | 17.0 | 22.7 |

### Languages where each system has the fewest serious errors, ties counted for each

| System | Languages |
|---|---|
| OpenTalk | 5 |
| Haiku | 11 |
| 31B | 9 |
| Qwen | 0 |
| Tencent | 0 |
| Plain E4B | 0 |

## Every language in full

### Arabic

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 5.3 | 4.7 | 1.3 | 15.3 | 11.3 | 9.3 |
| Translations with a serious error, % | 5.3 | 4.7 | 1.3 | 14.7 | 11.3 | 9.3 |
| Critical errors per 100 | 2.0 | 0.7 | 0.0 | 2.0 | 2.7 | 1.3 |
| Major errors per 100 | 3.3 | 4.0 | 1.3 | 13.3 | 8.7 | 8.0 |
| Minor errors per 100 | 10.7 | 8.7 | 4.0 | 22.7 | 16.7 | 16.7 |
| Cosmetic errors per 100 | 14.0 | 11.3 | 11.3 | 33.3 | 28.0 | 22.0 |
| All errors per 100 | 32.0 | 26.0 | 27.3 | 77.3 | 58.7 | 50.7 |
| Error-free translations, % | 72.0 | 77.3 | 74.0 | 42.0 | 50.0 | 54.7 |
| Fluency, mean of 1 to 5 | 4.41 | 4.61 | 4.37 | 4.21 | 4.41 | 4.17 |
| Fluency 5, % | 49.3 | 67.3 | 43.3 | 42.7 | 46.7 | 35.3 |
| Fluency 4, % | 43.3 | 26.0 | 50.0 | 38.7 | 48.0 | 46.7 |
| Fluency 3, % | 6.7 | 6.7 | 6.7 | 16.0 | 5.3 | 17.3 |
| Fluency 2 or 1, % | 0.7 | 0.0 | 0.0 | 2.7 | 0.0 | 0.7 |
| Key items wrong per 100 | 6.0 | 4.0 | 1.3 | 14.0 | 10.7 | 10.0 |
| Translations with a key item wrong, % | 6.0 | 4.0 | 1.3 | 13.3 | 10.0 | 10.0 |
| Share of key items wrong, % | 1.58 | 1.05 | 0.35 | 3.69 | 2.81 | 2.64 |
| Unusable, count | 0 | 0 | 0 | 1 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.67 | 0.00 | 0.00 |
| Composite, mean | 95.08 | 96.64 | 96.38 | 91.04 | 93.14 | 92.77 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 89.52 | 92.06 | 94.00 | 75.29 | 79.62 | 84.33 |
| Composite, double serious penalty | 93.83 | 95.79 | 96.20 | 88.34 | 90.59 | 91.01 |
| Serious fidelity errors per 100 | 1.3 | 2.0 | 1.3 | 10.7 | 6.0 | 5.3 |
| Serious logic errors per 100 | 2.0 | 0.0 | 0.0 | 0.0 | 2.0 | 2.0 |
| Serious numbers errors per 100 | 0.7 | 0.0 | 0.0 | 1.3 | 2.7 | 0.0 |
| Serious comprehension errors per 100 | 1.3 | 2.7 | 0.0 | 3.3 | 0.7 | 2.0 |
| All fidelity errors per 100 | 12.7 | 10.0 | 8.7 | 36.7 | 31.3 | 14.7 |
| All logic errors per 100 | 2.7 | 1.3 | 0.7 | 2.0 | 4.7 | 2.7 |
| All numbers errors per 100 | 0.7 | 0.0 | 1.3 | 4.0 | 7.3 | 2.7 |
| All comprehension errors per 100 | 14.0 | 13.3 | 6.0 | 28.7 | 12.7 | 28.0 |
| All form errors per 100 | 2.0 | 1.3 | 10.7 | 6.0 | 2.7 | 2.7 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -0.7 (-5.3 to +4.0) | -4.0 (-8.0 to +0.0) | +10.0 (+3.3 to +16.0) clear | +6.0 (+0.7 to +12.7) clear | +4.0 (-0.7 to +8.7) |
| Critical errors per 100 | -1.3 (-4.0 to +1.3) | -2.0 (-4.0 to +0.0) | +0.0 (-3.3 to +2.7) | +0.7 (-2.7 to +4.0) | -0.7 (-3.3 to +1.3) |
| Error-free, points | +5.3 (-3.3 to +14.0) | +2.0 (-8.7 to +10.7) | -30.0 (-40.0 to -20.7) clear | -22.0 (-31.3 to -14.7) clear | -17.3 (-26.7 to -8.7) clear |
| Fluency | +0.19 (+0.05 to +0.32) clear | -0.05 (-0.19 to +0.09) | -0.20 (-0.35 to -0.06) clear | +0.00 (-0.13 to +0.14) | -0.25 (-0.37 to -0.11) clear |
| Key items wrong per 100 | -2.0 (-6.0 to +2.7) | -4.7 (-8.7 to -1.3) clear | +8.0 (+1.3 to +14.7) clear | +4.7 (-0.7 to +11.3) | +4.0 (-0.7 to +9.3) |
| Composite | +1.56 (-0.11 to +3.13) | +1.30 (-0.18 to +2.96) | -4.04 (-6.13 to -1.66) clear | -1.94 (-4.21 to +0.03) | -2.32 (-4.00 to -0.53) clear |

### Brazilian Portuguese

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 3.3 | 4.0 | 4.0 | 6.0 | 11.3 | 9.3 |
| Translations with a serious error, % | 3.3 | 4.0 | 4.0 | 5.3 | 10.7 | 8.7 |
| Critical errors per 100 | 0.0 | 0.7 | 1.3 | 0.7 | 6.0 | 2.7 |
| Major errors per 100 | 3.3 | 3.3 | 2.7 | 5.3 | 5.3 | 6.7 |
| Minor errors per 100 | 4.7 | 4.0 | 2.7 | 8.7 | 10.0 | 7.3 |
| Cosmetic errors per 100 | 6.0 | 8.7 | 12.7 | 21.3 | 17.3 | 18.0 |
| All errors per 100 | 14.7 | 18.0 | 32.0 | 36.7 | 39.3 | 37.3 |
| Error-free translations, % | 86.7 | 84.0 | 70.7 | 66.0 | 68.0 | 66.7 |
| Fluency, mean of 1 to 5 | 4.66 | 4.80 | 4.49 | 4.62 | 4.70 | 4.30 |
| Fluency 5, % | 69.3 | 83.3 | 56.0 | 67.3 | 72.7 | 45.3 |
| Fluency 4, % | 28.0 | 14.0 | 36.7 | 28.0 | 25.3 | 41.3 |
| Fluency 3, % | 2.0 | 2.0 | 7.3 | 4.0 | 1.3 | 11.3 |
| Fluency 2 or 1, % | 0.7 | 0.7 | 0.0 | 0.7 | 0.7 | 2.0 |
| Key items wrong per 100 | 4.0 | 4.0 | 4.0 | 6.0 | 12.7 | 10.0 |
| Translations with a key item wrong, % | 4.0 | 4.0 | 4.0 | 4.7 | 11.3 | 8.7 |
| Share of key items wrong, % | 1.05 | 1.05 | 1.05 | 1.58 | 3.34 | 2.64 |
| Unusable, count | 0 | 0 | 0 | 0 | 1 | 1 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.67 | 0.67 |
| Composite, mean | 97.38 | 97.81 | 95.96 | 96.11 | 94.13 | 93.30 |
| Composite, median | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 95.00 |
| Composite, lowest 10% cut-off | 94.36 | 95.00 | 90.00 | 92.22 | 76.18 | 86.00 |
| Composite, double serious penalty | 96.84 | 97.06 | 94.99 | 95.23 | 91.46 | 91.64 |
| Serious fidelity errors per 100 | 2.0 | 0.7 | 0.7 | 4.0 | 4.7 | 5.3 |
| Serious logic errors per 100 | 1.3 | 0.0 | 1.3 | 0.7 | 2.7 | 0.7 |
| Serious numbers errors per 100 | 0.0 | 2.0 | 2.0 | 1.3 | 3.3 | 3.3 |
| Serious comprehension errors per 100 | 0.0 | 1.3 | 0.0 | 0.0 | 0.7 | 0.0 |
| All fidelity errors per 100 | 4.7 | 6.7 | 2.7 | 18.7 | 18.0 | 10.7 |
| All logic errors per 100 | 1.3 | 0.7 | 1.3 | 1.3 | 4.0 | 0.7 |
| All numbers errors per 100 | 2.0 | 4.0 | 3.3 | 6.0 | 5.3 | 4.7 |
| All comprehension errors per 100 | 6.0 | 5.3 | 12.0 | 10.0 | 11.3 | 18.7 |
| All form errors per 100 | 0.7 | 1.3 | 12.7 | 0.7 | 0.7 | 2.7 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | +0.7 (-3.3 to +4.7) | +0.7 (-4.0 to +5.3) | +2.7 (-1.3 to +7.3) | +8.0 (+2.0 to +14.7) clear | +6.0 (+1.3 to +11.3) clear |
| Critical errors per 100 | +0.7 (+0.0 to +2.0) | +1.3 (+0.0 to +3.3) | +0.7 (+0.0 to +2.0) | +6.0 (+2.7 to +10.0) clear | +2.7 (+0.0 to +6.0) |
| Error-free, points | -2.7 (-10.0 to +4.7) | -16.0 (-24.7 to -7.3) clear | -20.7 (-30.0 to -12.7) clear | -18.7 (-27.3 to -10.0) clear | -20.0 (-27.3 to -12.0) clear |
| Fluency | +0.14 (+0.02 to +0.27) clear | -0.17 (-0.29 to -0.05) clear | -0.04 (-0.17 to +0.09) | +0.04 (-0.09 to +0.18) | -0.36 (-0.50 to -0.23) clear |
| Key items wrong per 100 | +0.0 (-4.7 to +4.7) | +0.0 (-5.3 to +4.7) | +2.0 (-2.7 to +6.7) | +8.7 (+2.0 to +16.0) clear | +6.0 (+0.7 to +12.0) clear |
| Composite | +0.44 (-0.86 to +1.73) | -1.41 (-3.01 to -0.05) clear | -1.26 (-2.97 to +0.28) | -3.24 (-5.87 to -1.06) clear | -4.07 (-6.21 to -2.34) clear |

### Dutch

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 7.3 | 2.0 | 6.0 | 14.0 | 12.0 | 16.7 |
| Translations with a serious error, % | 6.7 | 2.0 | 6.0 | 13.3 | 11.3 | 15.3 |
| Critical errors per 100 | 0.0 | 0.7 | 0.7 | 2.0 | 2.0 | 2.7 |
| Major errors per 100 | 7.3 | 1.3 | 5.3 | 12.0 | 10.0 | 14.0 |
| Minor errors per 100 | 8.0 | 10.7 | 11.3 | 24.7 | 11.3 | 21.3 |
| Cosmetic errors per 100 | 23.3 | 19.3 | 14.0 | 27.3 | 22.7 | 28.7 |
| All errors per 100 | 44.7 | 34.0 | 44.0 | 71.3 | 49.3 | 70.0 |
| Error-free translations, % | 60.7 | 70.0 | 61.3 | 49.3 | 59.3 | 47.3 |
| Fluency, mean of 1 to 5 | 4.27 | 4.53 | 4.27 | 4.15 | 4.43 | 3.98 |
| Fluency 5, % | 39.3 | 62.7 | 40.7 | 40.7 | 56.7 | 30.7 |
| Fluency 4, % | 49.3 | 28.0 | 48.0 | 36.0 | 30.0 | 42.7 |
| Fluency 3, % | 10.0 | 9.3 | 9.3 | 21.3 | 12.7 | 20.7 |
| Fluency 2 or 1, % | 1.3 | 0.0 | 2.0 | 2.0 | 0.7 | 6.0 |
| Key items wrong per 100 | 6.0 | 2.7 | 5.3 | 15.3 | 12.7 | 17.3 |
| Translations with a key item wrong, % | 6.0 | 2.7 | 5.3 | 14.0 | 12.0 | 16.0 |
| Share of key items wrong, % | 1.58 | 0.70 | 1.41 | 4.04 | 3.34 | 4.57 |
| Unusable, count | 0 | 0 | 0 | 0 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Composite, mean | 94.25 | 96.57 | 94.40 | 91.01 | 93.46 | 89.44 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 98.23 | 95.00 |
| Composite, lowest 10% cut-off | 88.00 | 90.00 | 87.22 | 70.00 | 83.01 | 68.33 |
| Composite, double serious penalty | 92.95 | 96.17 | 93.24 | 88.58 | 91.39 | 86.01 |
| Serious fidelity errors per 100 | 5.3 | 1.3 | 3.3 | 7.3 | 6.7 | 7.3 |
| Serious logic errors per 100 | 1.3 | 0.0 | 0.0 | 1.3 | 2.0 | 2.0 |
| Serious numbers errors per 100 | 0.0 | 0.7 | 2.0 | 1.3 | 2.0 | 4.0 |
| Serious comprehension errors per 100 | 0.7 | 0.0 | 0.7 | 4.0 | 1.3 | 3.3 |
| All fidelity errors per 100 | 10.0 | 12.0 | 7.3 | 20.7 | 24.0 | 18.0 |
| All logic errors per 100 | 2.7 | 0.7 | 1.3 | 2.7 | 4.0 | 3.3 |
| All numbers errors per 100 | 0.7 | 2.7 | 3.3 | 5.3 | 3.3 | 8.0 |
| All comprehension errors per 100 | 25.3 | 16.7 | 19.3 | 37.3 | 14.7 | 37.3 |
| All form errors per 100 | 6.0 | 2.0 | 12.7 | 5.3 | 3.3 | 3.3 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -5.3 (-11.3 to +0.0) | -1.3 (-7.3 to +4.0) | +6.7 (+0.0 to +12.7) | +4.7 (-2.0 to +12.0) | +9.3 (+1.3 to +17.3) clear |
| Critical errors per 100 | +0.7 (+0.0 to +2.0) | +0.7 (+0.0 to +2.0) | +2.0 (+0.0 to +4.7) | +2.0 (+0.0 to +4.7) | +2.7 (+0.7 to +6.0) clear |
| Error-free, points | +9.3 (+0.0 to +18.0) | +0.7 (-10.7 to +10.7) | -11.3 (-22.0 to -0.7) clear | -1.3 (-11.3 to +8.7) | -13.3 (-22.0 to -4.7) clear |
| Fluency | +0.27 (+0.13 to +0.40) clear | +0.01 (-0.13 to +0.14) | -0.11 (-0.26 to +0.03) | +0.16 (+0.00 to +0.31) | -0.29 (-0.43 to -0.15) clear |
| Key items wrong per 100 | -3.3 (-8.0 to +1.3) | -0.7 (-6.0 to +4.0) | +9.3 (+2.7 to +16.0) clear | +6.7 (+0.0 to +13.3) | +11.3 (+4.0 to +19.3) clear |
| Composite | +2.32 (+0.68 to +3.83) clear | +0.15 (-1.54 to +1.96) | -3.24 (-5.18 to -1.04) clear | -0.79 (-3.27 to +1.11) | -4.81 (-7.36 to -2.40) clear |

### European Spanish

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 4.0 | 3.3 | 2.7 | 6.7 | 9.3 | 4.7 |
| Translations with a serious error, % | 4.0 | 3.3 | 2.7 | 6.0 | 9.3 | 4.7 |
| Critical errors per 100 | 0.7 | 0.7 | 0.7 | 0.7 | 1.3 | 2.0 |
| Major errors per 100 | 3.3 | 2.7 | 2.0 | 6.0 | 8.0 | 2.7 |
| Minor errors per 100 | 5.3 | 2.0 | 6.0 | 3.3 | 8.0 | 8.0 |
| Cosmetic errors per 100 | 10.0 | 8.7 | 12.7 | 24.7 | 17.3 | 18.0 |
| All errors per 100 | 21.3 | 14.7 | 38.0 | 38.0 | 38.0 | 32.0 |
| Error-free translations, % | 80.7 | 85.3 | 66.0 | 64.0 | 67.3 | 71.3 |
| Fluency, mean of 1 to 5 | 4.63 | 4.76 | 4.51 | 4.59 | 4.69 | 4.43 |
| Fluency 5, % | 69.3 | 79.3 | 59.3 | 66.0 | 73.3 | 53.3 |
| Fluency 4, % | 24.7 | 17.3 | 32.0 | 28.0 | 22.0 | 36.0 |
| Fluency 3, % | 6.0 | 3.3 | 8.7 | 5.3 | 4.7 | 10.7 |
| Fluency 2 or 1, % | 0.0 | 0.0 | 0.0 | 0.7 | 0.0 | 0.0 |
| Key items wrong per 100 | 3.3 | 3.3 | 3.3 | 6.7 | 9.3 | 5.3 |
| Translations with a key item wrong, % | 3.3 | 3.3 | 3.3 | 6.0 | 8.7 | 5.3 |
| Share of key items wrong, % | 0.88 | 0.88 | 0.88 | 1.76 | 2.46 | 1.41 |
| Unusable, count | 0 | 0 | 0 | 0 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Composite, mean | 97.00 | 97.93 | 96.47 | 96.02 | 95.65 | 95.31 |
| Composite, median | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 99.32 |
| Composite, lowest 10% cut-off | 94.00 | 94.00 | 90.00 | 90.00 | 87.44 | 89.00 |
| Composite, double serious penalty | 96.28 | 97.34 | 95.89 | 94.75 | 93.83 | 94.17 |
| Serious fidelity errors per 100 | 1.3 | 0.7 | 0.7 | 3.3 | 6.0 | 2.7 |
| Serious logic errors per 100 | 0.0 | 0.0 | 0.0 | 0.7 | 0.7 | 0.7 |
| Serious numbers errors per 100 | 1.3 | 2.7 | 1.3 | 2.0 | 2.7 | 1.3 |
| Serious comprehension errors per 100 | 1.3 | 0.0 | 0.7 | 0.7 | 0.0 | 0.0 |
| All fidelity errors per 100 | 5.3 | 3.3 | 4.7 | 18.0 | 16.7 | 8.7 |
| All logic errors per 100 | 1.3 | 0.7 | 0.0 | 0.7 | 3.3 | 0.7 |
| All numbers errors per 100 | 2.0 | 4.0 | 2.0 | 2.7 | 3.3 | 2.7 |
| All comprehension errors per 100 | 10.7 | 6.0 | 14.7 | 13.3 | 11.3 | 18.7 |
| All form errors per 100 | 2.0 | 0.7 | 16.7 | 3.3 | 3.3 | 1.3 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -0.7 (-4.0 to +2.7) | -1.3 (-4.7 to +2.0) | +2.7 (-2.0 to +8.7) | +5.3 (+0.0 to +11.3) | +0.7 (-3.3 to +4.7) |
| Critical errors per 100 | +0.0 (-2.0 to +2.0) | +0.0 (-2.0 to +2.0) | +0.0 (-2.0 to +2.0) | +0.7 (-1.3 to +3.3) | +1.3 (-1.3 to +4.0) |
| Error-free, points | +4.7 (-2.7 to +10.7) | -14.7 (-23.3 to -6.0) clear | -16.7 (-26.0 to -7.3) clear | -13.3 (-22.0 to -5.3) clear | -9.3 (-17.3 to -0.7) clear |
| Fluency | +0.13 (+0.03 to +0.22) clear | -0.13 (-0.27 to +0.01) | -0.04 (-0.17 to +0.10) | +0.05 (-0.07 to +0.19) | -0.21 (-0.34 to -0.07) clear |
| Key items wrong per 100 | +0.0 (-3.3 to +3.3) | +0.0 (-3.3 to +3.3) | +3.3 (-1.3 to +8.7) | +6.0 (+0.7 to +11.3) clear | +2.0 (-2.0 to +6.0) |
| Composite | +0.93 (-0.03 to +1.93) | -0.53 (-1.71 to +0.70) | -0.98 (-2.66 to +0.67) | -1.35 (-2.94 to +0.25) | -1.69 (-3.26 to -0.07) clear |

### French

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 2.0 | 4.0 | 3.3 | 6.0 | 10.0 | 6.0 |
| Translations with a serious error, % | 2.0 | 4.0 | 3.3 | 5.3 | 9.3 | 6.0 |
| Critical errors per 100 | 0.7 | 2.0 | 0.0 | 0.7 | 1.3 | 2.7 |
| Major errors per 100 | 1.3 | 2.0 | 3.3 | 5.3 | 8.7 | 3.3 |
| Minor errors per 100 | 10.0 | 5.3 | 6.0 | 11.3 | 13.3 | 11.3 |
| Cosmetic errors per 100 | 12.0 | 11.3 | 11.3 | 25.3 | 30.7 | 20.0 |
| All errors per 100 | 26.7 | 20.7 | 33.3 | 44.7 | 55.3 | 40.7 |
| Error-free translations, % | 75.3 | 81.3 | 69.3 | 61.3 | 58.0 | 62.7 |
| Fluency, mean of 1 to 5 | 4.41 | 4.78 | 4.49 | 4.69 | 4.61 | 4.28 |
| Fluency 5, % | 52.0 | 80.7 | 56.0 | 74.7 | 68.7 | 42.7 |
| Fluency 4, % | 38.0 | 16.7 | 37.3 | 20.7 | 23.3 | 42.7 |
| Fluency 3, % | 9.3 | 2.7 | 6.7 | 4.0 | 8.0 | 14.7 |
| Fluency 2 or 1, % | 0.7 | 0.0 | 0.0 | 0.7 | 0.0 | 0.0 |
| Key items wrong per 100 | 2.0 | 4.0 | 3.3 | 6.0 | 8.7 | 6.7 |
| Translations with a key item wrong, % | 2.0 | 4.0 | 3.3 | 4.7 | 8.7 | 6.7 |
| Share of key items wrong, % | 0.53 | 1.05 | 0.88 | 1.58 | 2.28 | 1.76 |
| Unusable, count | 0 | 0 | 0 | 0 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Composite, mean | 96.06 | 97.55 | 96.42 | 96.40 | 94.83 | 93.81 |
| Composite, median | 99.31 | 100.00 | 100.00 | 100.00 | 99.38 | 95.00 |
| Composite, lowest 10% cut-off | 90.00 | 94.00 | 91.00 | 89.35 | 85.42 | 86.00 |
| Composite, double serious penalty | 95.58 | 96.66 | 95.87 | 95.53 | 92.79 | 92.19 |
| Serious fidelity errors per 100 | 1.3 | 2.7 | 1.3 | 4.7 | 8.0 | 1.3 |
| Serious logic errors per 100 | 0.7 | 0.7 | 0.7 | 0.0 | 0.7 | 2.7 |
| Serious numbers errors per 100 | 0.0 | 0.7 | 1.3 | 1.3 | 1.3 | 2.0 |
| Serious comprehension errors per 100 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| All fidelity errors per 100 | 5.3 | 6.0 | 6.7 | 20.7 | 28.0 | 9.3 |
| All logic errors per 100 | 2.0 | 2.0 | 0.7 | 4.0 | 2.7 | 2.7 |
| All numbers errors per 100 | 3.3 | 2.0 | 2.0 | 3.3 | 3.3 | 3.3 |
| All comprehension errors per 100 | 13.3 | 10.7 | 11.3 | 14.7 | 20.0 | 22.0 |
| All form errors per 100 | 2.7 | 0.0 | 12.7 | 2.0 | 1.3 | 3.3 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | +2.0 (-2.0 to +6.0) | +1.3 (-2.0 to +5.3) | +4.0 (-0.7 to +8.7) | +8.0 (+2.7 to +14.0) clear | +4.0 (+0.0 to +8.7) |
| Critical errors per 100 | +1.3 (-0.7 to +4.0) | -0.7 (-2.0 to +0.0) | +0.0 (-2.0 to +1.3) | +0.7 (-1.3 to +2.7) | +2.0 (-0.7 to +5.3) |
| Error-free, points | +6.0 (-2.0 to +13.3) | -6.0 (-16.0 to +2.0) | -14.0 (-24.0 to -4.7) clear | -17.3 (-28.0 to -8.0) clear | -12.7 (-21.3 to -4.7) clear |
| Fluency | +0.37 (+0.23 to +0.49) clear | +0.08 (-0.06 to +0.21) | +0.28 (+0.15 to +0.41) clear | +0.19 (+0.06 to +0.34) clear | -0.13 (-0.26 to -0.02) clear |
| Key items wrong per 100 | +2.0 (-2.0 to +6.0) | +1.3 (-2.0 to +5.3) | +4.0 (-0.7 to +9.3) | +6.7 (+2.0 to +11.3) clear | +4.7 (+0.0 to +9.3) |
| Composite | +1.49 (+0.08 to +3.01) clear | +0.36 (-1.09 to +1.83) | +0.34 (-1.20 to +2.09) | -1.23 (-2.86 to +0.54) | -2.25 (-4.01 to -0.45) clear |

### German

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 6.7 | 4.0 | 5.3 | 10.7 | 14.7 | 15.3 |
| Translations with a serious error, % | 6.0 | 4.0 | 5.3 | 10.7 | 14.0 | 14.7 |
| Critical errors per 100 | 0.0 | 0.7 | 1.3 | 3.3 | 3.3 | 5.3 |
| Major errors per 100 | 6.7 | 3.3 | 4.0 | 7.3 | 11.3 | 10.0 |
| Minor errors per 100 | 9.3 | 10.0 | 6.7 | 17.3 | 17.3 | 24.7 |
| Cosmetic errors per 100 | 22.0 | 6.7 | 12.7 | 26.7 | 22.7 | 30.7 |
| All errors per 100 | 40.0 | 22.0 | 42.7 | 57.3 | 58.0 | 72.7 |
| Error-free translations, % | 68.0 | 79.3 | 61.3 | 53.3 | 51.3 | 44.7 |
| Fluency, mean of 1 to 5 | 4.29 | 4.66 | 4.29 | 4.35 | 4.47 | 3.89 |
| Fluency 5, % | 44.7 | 71.3 | 44.0 | 50.7 | 58.7 | 25.3 |
| Fluency 4, % | 41.3 | 24.0 | 41.3 | 34.7 | 30.7 | 42.0 |
| Fluency 3, % | 12.0 | 4.0 | 14.0 | 13.3 | 10.0 | 29.3 |
| Fluency 2 or 1, % | 2.0 | 0.7 | 0.7 | 1.3 | 0.7 | 3.3 |
| Key items wrong per 100 | 6.0 | 5.3 | 5.3 | 12.7 | 15.3 | 17.3 |
| Translations with a key item wrong, % | 5.3 | 5.3 | 5.3 | 12.0 | 15.3 | 14.7 |
| Share of key items wrong, % | 1.58 | 1.41 | 1.41 | 3.34 | 4.04 | 4.57 |
| Unusable, count | 0 | 0 | 0 | 0 | 0 | 1 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.67 |
| Composite, mean | 94.50 | 96.75 | 94.62 | 92.82 | 92.64 | 88.58 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 98.95 | 94.29 |
| Composite, lowest 10% cut-off | 86.53 | 91.00 | 89.00 | 81.00 | 73.33 | 68.33 |
| Composite, double serious penalty | 93.46 | 95.94 | 93.41 | 90.37 | 89.50 | 85.37 |
| Serious fidelity errors per 100 | 2.7 | 2.0 | 2.0 | 7.3 | 8.0 | 8.0 |
| Serious logic errors per 100 | 0.7 | 0.7 | 0.7 | 2.0 | 2.0 | 0.7 |
| Serious numbers errors per 100 | 0.7 | 1.3 | 2.7 | 1.3 | 2.7 | 4.7 |
| Serious comprehension errors per 100 | 2.7 | 0.0 | 0.0 | 0.0 | 2.0 | 2.0 |
| All fidelity errors per 100 | 12.7 | 8.7 | 4.0 | 26.0 | 24.7 | 16.7 |
| All logic errors per 100 | 3.3 | 0.7 | 0.7 | 4.7 | 8.0 | 2.7 |
| All numbers errors per 100 | 2.7 | 4.0 | 5.3 | 4.0 | 6.0 | 9.3 |
| All comprehension errors per 100 | 19.3 | 7.3 | 14.7 | 20.0 | 16.0 | 42.0 |
| All form errors per 100 | 2.0 | 1.3 | 18.0 | 2.7 | 3.3 | 2.0 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -2.7 (-8.7 to +2.7) | -1.3 (-7.3 to +4.7) | +4.0 (-2.0 to +10.0) | +8.0 (+0.7 to +16.0) clear | +8.7 (+1.3 to +16.0) clear |
| Critical errors per 100 | +0.7 (+0.0 to +2.0) | +1.3 (+0.0 to +3.3) | +3.3 (+0.7 to +6.7) clear | +3.3 (+0.7 to +6.7) clear | +5.3 (+2.0 to +9.3) clear |
| Error-free, points | +11.3 (+2.7 to +20.0) clear | -6.7 (-17.3 to +3.3) | -14.7 (-24.7 to -5.3) clear | -16.7 (-26.7 to -6.7) clear | -23.3 (-32.0 to -14.7) clear |
| Fluency | +0.37 (+0.21 to +0.53) clear | +0.00 (-0.15 to +0.15) | +0.06 (-0.11 to +0.21) | +0.19 (+0.03 to +0.33) clear | -0.39 (-0.55 to -0.23) clear |
| Key items wrong per 100 | -0.7 (-6.0 to +4.7) | -0.7 (-6.0 to +5.3) | +6.7 (+0.7 to +14.0) clear | +9.3 (+3.3 to +16.7) clear | +11.3 (+4.0 to +19.3) clear |
| Composite | +2.25 (+0.69 to +3.87) clear | +0.12 (-1.75 to +2.21) | -1.67 (-3.88 to +0.47) | -1.85 (-4.30 to +0.12) | -5.92 (-8.73 to -3.26) clear |

### Hindi

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 4.7 | 2.7 | 1.3 | 28.7 | 15.3 | 12.7 |
| Translations with a serious error, % | 4.7 | 2.7 | 1.3 | 24.0 | 13.3 | 12.0 |
| Critical errors per 100 | 0.7 | 0.7 | 0.7 | 4.7 | 2.0 | 2.7 |
| Major errors per 100 | 4.0 | 2.0 | 0.7 | 24.0 | 13.3 | 10.0 |
| Minor errors per 100 | 11.3 | 6.7 | 9.3 | 32.7 | 22.0 | 14.7 |
| Cosmetic errors per 100 | 11.3 | 9.3 | 16.0 | 29.3 | 26.0 | 18.7 |
| All errors per 100 | 28.7 | 19.3 | 50.0 | 96.7 | 63.3 | 46.7 |
| Error-free translations, % | 74.7 | 81.3 | 55.3 | 30.7 | 50.0 | 62.0 |
| Fluency, mean of 1 to 5 | 4.44 | 4.61 | 3.95 | 3.62 | 4.05 | 3.97 |
| Fluency 5, % | 51.3 | 61.3 | 23.3 | 14.0 | 18.7 | 20.0 |
| Fluency 4, % | 42.7 | 38.0 | 49.3 | 42.0 | 68.7 | 58.0 |
| Fluency 3, % | 4.7 | 0.7 | 26.0 | 36.7 | 12.0 | 20.7 |
| Fluency 2 or 1, % | 1.3 | 0.0 | 1.3 | 7.3 | 0.7 | 1.3 |
| Key items wrong per 100 | 4.7 | 3.3 | 1.3 | 27.3 | 17.3 | 14.7 |
| Translations with a key item wrong, % | 4.7 | 3.3 | 1.3 | 22.0 | 14.0 | 12.7 |
| Share of key items wrong, % | 1.23 | 0.88 | 0.35 | 7.21 | 4.57 | 3.87 |
| Unusable, count | 0 | 0 | 0 | 3 | 0 | 1 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 2.00 | 0.00 | 0.67 |
| Composite, mean | 95.73 | 97.07 | 93.90 | 83.99 | 90.32 | 90.30 |
| Composite, median | 99.00 | 100.00 | 95.00 | 90.50 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 90.00 | 94.33 | 89.00 | 61.92 | 73.64 | 73.80 |
| Composite, double serious penalty | 94.94 | 96.54 | 93.55 | 79.32 | 87.29 | 88.01 |
| Serious fidelity errors per 100 | 1.3 | 1.3 | 0.7 | 13.3 | 10.7 | 4.7 |
| Serious logic errors per 100 | 0.7 | 0.7 | 0.0 | 2.0 | 1.3 | 2.7 |
| Serious numbers errors per 100 | 2.0 | 0.7 | 0.7 | 6.0 | 3.3 | 4.7 |
| Serious comprehension errors per 100 | 0.7 | 0.0 | 0.0 | 7.3 | 0.0 | 0.7 |
| All fidelity errors per 100 | 8.0 | 6.7 | 6.7 | 32.0 | 41.3 | 12.0 |
| All logic errors per 100 | 5.3 | 3.3 | 0.7 | 7.3 | 5.3 | 4.7 |
| All numbers errors per 100 | 4.7 | 2.7 | 3.3 | 7.3 | 8.0 | 10.7 |
| All comprehension errors per 100 | 9.3 | 6.0 | 16.0 | 44.0 | 8.7 | 18.7 |
| All form errors per 100 | 1.3 | 0.7 | 23.3 | 6.0 | 0.0 | 0.7 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -2.0 (-6.7 to +2.7) | -3.3 (-7.3 to +0.7) | +24.0 (+14.7 to +33.3) clear | +10.7 (+3.3 to +18.0) clear | +8.0 (+2.0 to +13.3) clear |
| Critical errors per 100 | +0.0 (-2.0 to +2.0) | +0.0 (-1.3 to +2.0) | +4.0 (+0.7 to +8.0) clear | +1.3 (-1.3 to +4.0) | +2.0 (-0.7 to +5.3) |
| Error-free, points | +6.7 (-2.7 to +15.3) | -19.3 (-30.0 to -8.7) clear | -44.0 (-54.0 to -34.0) clear | -24.7 (-36.0 to -13.3) clear | -12.7 (-21.3 to -3.3) clear |
| Fluency | +0.17 (+0.04 to +0.27) clear | -0.49 (-0.65 to -0.35) clear | -0.82 (-0.97 to -0.68) clear | -0.39 (-0.52 to -0.25) clear | -0.47 (-0.61 to -0.35) clear |
| Key items wrong per 100 | -1.3 (-6.0 to +3.3) | -3.3 (-7.3 to +0.0) | +22.7 (+13.3 to +32.7) clear | +12.7 (+5.3 to +20.7) clear | +10.0 (+3.3 to +16.7) clear |
| Composite | +1.34 (-0.09 to +2.67) | -1.83 (-3.27 to -0.39) clear | -11.75 (-14.74 to -8.80) clear | -5.41 (-7.67 to -3.19) clear | -5.43 (-7.94 to -3.31) clear |

### Indonesian

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 5.3 | 2.0 | 4.7 | 7.3 | 10.7 | 6.7 |
| Translations with a serious error, % | 5.3 | 2.0 | 4.7 | 6.7 | 10.0 | 6.7 |
| Critical errors per 100 | 0.0 | 0.0 | 0.7 | 2.7 | 1.3 | 1.3 |
| Major errors per 100 | 5.3 | 2.0 | 4.0 | 4.7 | 9.3 | 5.3 |
| Minor errors per 100 | 6.0 | 2.7 | 4.0 | 8.0 | 13.3 | 12.0 |
| Cosmetic errors per 100 | 10.7 | 9.3 | 14.7 | 14.0 | 19.3 | 12.7 |
| All errors per 100 | 22.7 | 16.7 | 40.7 | 31.3 | 44.0 | 32.7 |
| Error-free translations, % | 78.0 | 83.3 | 62.0 | 71.3 | 61.3 | 68.0 |
| Fluency, mean of 1 to 5 | 4.51 | 4.69 | 4.25 | 4.62 | 4.61 | 4.37 |
| Fluency 5, % | 56.0 | 70.7 | 38.0 | 66.7 | 64.0 | 45.3 |
| Fluency 4, % | 38.7 | 28.0 | 49.3 | 28.7 | 32.7 | 47.3 |
| Fluency 3, % | 5.3 | 1.3 | 12.0 | 4.7 | 3.3 | 6.7 |
| Fluency 2 or 1, % | 0.0 | 0.0 | 0.7 | 0.0 | 0.0 | 0.7 |
| Key items wrong per 100 | 5.3 | 1.3 | 4.7 | 6.7 | 11.3 | 8.7 |
| Translations with a key item wrong, % | 5.3 | 1.3 | 4.7 | 6.0 | 10.7 | 8.0 |
| Share of key items wrong, % | 1.41 | 0.35 | 1.23 | 1.76 | 2.99 | 2.28 |
| Unusable, count | 0 | 0 | 0 | 1 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.67 | 0.00 | 0.00 |
| Composite, mean | 96.12 | 97.83 | 94.78 | 95.47 | 94.73 | 94.29 |
| Composite, median | 100.00 | 100.00 | 95.00 | 100.00 | 99.69 | 95.00 |
| Composite, lowest 10% cut-off | 90.00 | 95.00 | 89.32 | 91.00 | 87.56 | 86.00 |
| Composite, double serious penalty | 95.27 | 97.48 | 93.84 | 94.17 | 92.60 | 92.73 |
| Serious fidelity errors per 100 | 2.7 | 0.7 | 2.7 | 5.3 | 8.7 | 4.0 |
| Serious logic errors per 100 | 1.3 | 0.7 | 0.7 | 0.7 | 0.7 | 2.7 |
| Serious numbers errors per 100 | 0.7 | 0.7 | 0.7 | 0.7 | 1.3 | 0.0 |
| Serious comprehension errors per 100 | 0.7 | 0.0 | 0.7 | 0.7 | 0.0 | 0.0 |
| All fidelity errors per 100 | 9.3 | 5.3 | 5.3 | 18.0 | 24.7 | 12.7 |
| All logic errors per 100 | 2.0 | 1.3 | 2.0 | 2.0 | 5.3 | 4.0 |
| All numbers errors per 100 | 2.0 | 1.3 | 0.7 | 2.0 | 4.7 | 2.7 |
| All comprehension errors per 100 | 8.7 | 6.0 | 15.3 | 7.3 | 8.7 | 12.0 |
| All form errors per 100 | 0.7 | 2.7 | 17.3 | 2.0 | 0.7 | 1.3 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -3.3 (-6.7 to +0.0) | -0.7 (-5.3 to +4.0) | +2.0 (-3.3 to +8.0) | +5.3 (-0.7 to +12.0) | +1.3 (-2.7 to +6.0) |
| Critical errors per 100 | +0.0 (+0.0 to +0.0) | +0.7 (+0.0 to +2.0) | +2.7 (+0.0 to +6.0) | +1.3 (+0.0 to +3.3) | +1.3 (+0.0 to +3.3) |
| Error-free, points | +5.3 (-2.7 to +14.0) | -16.0 (-27.3 to -5.3) clear | -6.7 (-16.0 to +2.7) | -16.7 (-27.3 to -5.3) clear | -10.0 (-19.3 to -0.7) clear |
| Fluency | +0.19 (+0.09 to +0.29) clear | -0.26 (-0.41 to -0.11) clear | +0.11 (-0.01 to +0.25) | +0.10 (-0.03 to +0.23) | -0.13 (-0.25 to -0.01) clear |
| Key items wrong per 100 | -4.0 (-8.0 to +0.0) | -0.7 (-5.3 to +4.7) | +1.3 (-4.7 to +7.3) | +6.0 (+0.0 to +12.0) | +3.3 (-0.7 to +8.0) |
| Composite | +1.71 (+0.88 to +2.57) clear | -1.34 (-2.87 to +0.26) | -0.65 (-2.85 to +1.28) | -1.39 (-3.19 to +0.35) | -1.83 (-3.27 to -0.51) clear |

### Italian

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 1.3 | 4.0 | 1.3 | 7.3 | 10.7 | 6.7 |
| Translations with a serious error, % | 1.3 | 4.0 | 1.3 | 6.7 | 9.3 | 6.7 |
| Critical errors per 100 | 0.0 | 0.7 | 0.0 | 1.3 | 2.7 | 2.0 |
| Major errors per 100 | 1.3 | 3.3 | 1.3 | 6.0 | 8.0 | 4.7 |
| Minor errors per 100 | 7.3 | 4.0 | 8.7 | 14.7 | 16.7 | 22.7 |
| Cosmetic errors per 100 | 16.0 | 20.0 | 19.3 | 28.7 | 34.7 | 28.7 |
| All errors per 100 | 26.7 | 29.3 | 38.0 | 56.0 | 67.3 | 66.0 |
| Error-free translations, % | 78.0 | 72.0 | 66.0 | 56.7 | 46.7 | 46.7 |
| Fluency, mean of 1 to 5 | 4.43 | 4.59 | 4.43 | 4.39 | 4.31 | 3.98 |
| Fluency 5, % | 53.3 | 68.0 | 55.3 | 52.7 | 44.7 | 22.7 |
| Fluency 4, % | 37.3 | 23.3 | 33.3 | 34.7 | 41.3 | 54.0 |
| Fluency 3, % | 8.7 | 8.7 | 10.0 | 11.3 | 14.0 | 22.0 |
| Fluency 2 or 1, % | 0.7 | 0.0 | 1.3 | 1.3 | 0.0 | 1.3 |
| Key items wrong per 100 | 1.3 | 4.0 | 3.3 | 7.3 | 10.7 | 9.3 |
| Translations with a key item wrong, % | 1.3 | 3.3 | 3.3 | 6.0 | 9.3 | 8.7 |
| Share of key items wrong, % | 0.35 | 1.05 | 0.88 | 1.93 | 2.81 | 2.46 |
| Unusable, count | 0 | 0 | 0 | 0 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Composite, mean | 96.51 | 96.60 | 96.24 | 94.32 | 92.97 | 91.84 |
| Composite, median | 100.00 | 100.00 | 100.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 89.34 | 89.00 | 89.34 | 86.00 | 86.00 | 84.94 |
| Composite, double serious penalty | 96.29 | 95.75 | 96.02 | 92.84 | 90.80 | 90.36 |
| Serious fidelity errors per 100 | 0.7 | 1.3 | 0.0 | 4.7 | 6.7 | 4.0 |
| Serious logic errors per 100 | 0.7 | 0.0 | 0.0 | 0.7 | 1.3 | 0.7 |
| Serious numbers errors per 100 | 0.0 | 2.7 | 1.3 | 2.0 | 2.7 | 1.3 |
| Serious comprehension errors per 100 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.7 |
| All fidelity errors per 100 | 7.3 | 5.3 | 5.3 | 20.7 | 28.7 | 14.0 |
| All logic errors per 100 | 2.7 | 0.0 | 0.0 | 3.3 | 4.0 | 0.7 |
| All numbers errors per 100 | 0.7 | 4.0 | 3.3 | 3.3 | 4.0 | 4.0 |
| All comprehension errors per 100 | 14.0 | 18.7 | 20.7 | 23.3 | 25.3 | 39.3 |
| All form errors per 100 | 2.0 | 1.3 | 8.7 | 5.3 | 5.3 | 8.0 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | +2.7 (+0.0 to +6.0) | +0.0 (-2.7 to +2.7) | +6.0 (+2.0 to +11.3) clear | +9.3 (+4.7 to +14.7) clear | +5.3 (+1.3 to +10.0) clear |
| Critical errors per 100 | +0.7 (+0.0 to +2.0) | +0.0 (+0.0 to +0.0) | +1.3 (+0.0 to +3.3) | +2.7 (+0.7 to +6.0) clear | +2.0 (+0.0 to +4.7) |
| Error-free, points | -6.0 (-14.7 to +2.7) | -12.0 (-21.3 to -3.3) clear | -21.3 (-32.0 to -10.7) clear | -31.3 (-42.0 to -20.7) clear | -31.3 (-40.7 to -22.7) clear |
| Fluency | +0.16 (+0.03 to +0.30) clear | -0.01 (-0.13 to +0.13) | -0.05 (-0.19 to +0.11) | -0.13 (-0.28 to +0.03) | -0.45 (-0.57 to -0.33) clear |
| Key items wrong per 100 | +2.7 (-1.3 to +7.3) | +2.0 (-1.3 to +6.0) | +6.0 (+1.3 to +11.3) clear | +9.3 (+4.7 to +14.7) clear | +8.0 (+3.3 to +14.0) clear |
| Composite | +0.09 (-1.29 to +1.14) | -0.27 (-1.24 to +0.62) | -2.19 (-3.79 to -0.77) clear | -3.54 (-5.53 to -2.03) clear | -4.67 (-6.19 to -3.27) clear |

### Japanese

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 8.7 | 5.3 | 8.7 | 15.3 | 16.0 | 23.3 |
| Translations with a serious error, % | 7.3 | 4.7 | 8.7 | 14.0 | 14.7 | 22.0 |
| Critical errors per 100 | 2.7 | 1.3 | 1.3 | 4.7 | 6.0 | 3.3 |
| Major errors per 100 | 6.0 | 4.0 | 7.3 | 10.7 | 10.0 | 20.0 |
| Minor errors per 100 | 14.0 | 3.3 | 8.0 | 19.3 | 10.0 | 24.0 |
| Cosmetic errors per 100 | 13.3 | 10.7 | 14.7 | 30.0 | 16.7 | 22.7 |
| All errors per 100 | 37.3 | 20.0 | 38.0 | 66.0 | 42.7 | 72.0 |
| Error-free translations, % | 69.3 | 82.0 | 64.0 | 46.7 | 62.7 | 45.3 |
| Fluency, mean of 1 to 5 | 4.45 | 4.73 | 4.23 | 4.39 | 4.51 | 4.01 |
| Fluency 5, % | 56.0 | 74.0 | 35.3 | 44.7 | 55.3 | 28.0 |
| Fluency 4, % | 34.7 | 25.3 | 52.7 | 49.3 | 40.7 | 47.3 |
| Fluency 3, % | 8.0 | 0.7 | 11.3 | 6.0 | 4.0 | 22.7 |
| Fluency 2 or 1, % | 1.3 | 0.0 | 0.7 | 0.0 | 0.0 | 2.0 |
| Key items wrong per 100 | 10.0 | 4.7 | 8.0 | 16.7 | 13.3 | 23.3 |
| Translations with a key item wrong, % | 9.3 | 4.7 | 8.0 | 14.0 | 13.3 | 20.7 |
| Share of key items wrong, % | 2.64 | 1.23 | 2.11 | 4.39 | 3.51 | 6.15 |
| Unusable, count | 0 | 0 | 0 | 1 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.67 | 0.00 | 0.00 |
| Composite, mean | 94.45 | 97.13 | 93.37 | 91.06 | 92.37 | 88.02 |
| Composite, median | 100.00 | 100.00 | 95.00 | 95.00 | 96.00 | 94.67 |
| Composite, lowest 10% cut-off | 86.67 | 94.38 | 86.00 | 73.33 | 70.00 | 68.33 |
| Composite, double serious penalty | 92.99 | 96.04 | 91.49 | 88.09 | 88.62 | 83.62 |
| Serious fidelity errors per 100 | 4.0 | 1.3 | 4.0 | 10.7 | 8.0 | 8.0 |
| Serious logic errors per 100 | 2.7 | 1.3 | 1.3 | 2.0 | 3.3 | 6.0 |
| Serious numbers errors per 100 | 2.0 | 2.7 | 3.3 | 2.7 | 4.7 | 7.3 |
| Serious comprehension errors per 100 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 |
| All fidelity errors per 100 | 15.3 | 8.7 | 8.7 | 38.0 | 22.0 | 24.7 |
| All logic errors per 100 | 5.3 | 2.7 | 3.3 | 5.3 | 4.7 | 6.7 |
| All numbers errors per 100 | 4.0 | 3.3 | 4.7 | 7.3 | 8.7 | 12.0 |
| All comprehension errors per 100 | 11.3 | 4.7 | 14.7 | 14.0 | 7.3 | 26.7 |
| All form errors per 100 | 1.3 | 0.7 | 6.7 | 1.3 | 0.0 | 2.0 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -3.3 (-10.0 to +2.7) | +0.0 (-7.3 to +6.7) | +6.7 (+0.0 to +13.3) | +7.3 (-0.7 to +15.3) | +14.7 (+8.0 to +22.0) clear |
| Critical errors per 100 | -1.3 (-4.7 to +2.0) | -1.3 (-4.7 to +2.0) | +2.0 (-1.3 to +6.0) | +3.3 (-1.3 to +8.0) | +0.7 (-2.7 to +4.0) |
| Error-free, points | +12.7 (+3.3 to +22.0) clear | -5.3 (-16.0 to +4.0) | -22.7 (-34.0 to -12.0) clear | -6.7 (-17.3 to +3.3) | -24.0 (-32.7 to -15.3) clear |
| Fluency | +0.28 (+0.15 to +0.41) clear | -0.23 (-0.39 to -0.07) clear | -0.07 (-0.23 to +0.08) | +0.06 (-0.09 to +0.20) | -0.44 (-0.59 to -0.29) clear |
| Key items wrong per 100 | -5.3 (-12.0 to +0.7) | -2.0 (-8.7 to +5.3) | +6.7 (+0.0 to +14.7) | +3.3 (-4.0 to +11.3) | +13.3 (+6.0 to +21.3) clear |
| Composite | +2.68 (+0.52 to +4.83) clear | -1.08 (-3.57 to +1.38) | -3.39 (-6.13 to -1.00) clear | -2.08 (-4.88 to +0.82) | -6.42 (-8.58 to -4.32) clear |

### Korean

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 8.0 | 4.0 | 9.3 | 22.0 | 18.0 | 12.7 |
| Translations with a serious error, % | 8.0 | 4.0 | 8.7 | 21.3 | 16.7 | 12.0 |
| Critical errors per 100 | 1.3 | 1.3 | 0.7 | 5.3 | 4.0 | 3.3 |
| Major errors per 100 | 6.7 | 2.7 | 8.7 | 16.7 | 14.0 | 9.3 |
| Minor errors per 100 | 10.0 | 4.0 | 8.7 | 20.0 | 14.0 | 14.7 |
| Cosmetic errors per 100 | 18.0 | 10.0 | 16.0 | 25.3 | 22.7 | 23.3 |
| All errors per 100 | 38.7 | 19.3 | 45.3 | 70.0 | 55.3 | 50.7 |
| Error-free translations, % | 65.3 | 82.7 | 61.3 | 44.0 | 58.0 | 55.3 |
| Fluency, mean of 1 to 5 | 4.46 | 4.71 | 4.19 | 4.09 | 4.43 | 4.35 |
| Fluency 5, % | 52.0 | 74.7 | 36.0 | 29.3 | 50.0 | 44.0 |
| Fluency 4, % | 42.0 | 22.7 | 48.7 | 52.7 | 43.3 | 46.7 |
| Fluency 3, % | 6.0 | 1.3 | 13.3 | 15.3 | 6.0 | 9.3 |
| Fluency 2 or 1, % | 0.0 | 1.3 | 2.0 | 2.7 | 0.7 | 0.0 |
| Key items wrong per 100 | 8.7 | 3.3 | 8.7 | 23.3 | 20.7 | 14.7 |
| Translations with a key item wrong, % | 8.0 | 3.3 | 7.3 | 22.0 | 18.0 | 13.3 |
| Share of key items wrong, % | 2.28 | 0.88 | 2.28 | 6.15 | 5.45 | 3.87 |
| Unusable, count | 0 | 0 | 0 | 2 | 1 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 1.33 | 0.67 | 0.00 |
| Composite, mean | 94.70 | 97.23 | 93.05 | 87.75 | 91.14 | 92.17 |
| Composite, median | 95.00 | 100.00 | 95.00 | 94.33 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 86.67 | 95.00 | 86.00 | 64.33 | 68.33 | 73.33 |
| Composite, double serious penalty | 93.06 | 96.32 | 91.32 | 83.17 | 87.57 | 89.23 |
| Serious fidelity errors per 100 | 2.7 | 1.3 | 0.7 | 13.3 | 8.0 | 5.3 |
| Serious logic errors per 100 | 4.0 | 0.7 | 2.0 | 4.0 | 5.3 | 2.7 |
| Serious numbers errors per 100 | 1.3 | 2.0 | 5.3 | 3.3 | 4.7 | 4.7 |
| Serious comprehension errors per 100 | 0.0 | 0.0 | 1.3 | 1.3 | 0.0 | 0.0 |
| All fidelity errors per 100 | 10.7 | 8.0 | 7.3 | 33.3 | 29.3 | 20.7 |
| All logic errors per 100 | 6.7 | 2.7 | 4.7 | 8.0 | 8.0 | 6.0 |
| All numbers errors per 100 | 4.7 | 2.7 | 7.3 | 8.7 | 10.7 | 11.3 |
| All comprehension errors per 100 | 14.0 | 4.7 | 14.7 | 17.3 | 6.7 | 12.7 |
| All form errors per 100 | 2.7 | 1.3 | 11.3 | 2.7 | 0.7 | 0.0 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -4.0 (-9.3 to +0.7) | +1.3 (-5.3 to +7.3) | +14.0 (+6.0 to +21.3) clear | +10.0 (+1.3 to +18.7) clear | +4.7 (-1.3 to +10.7) |
| Critical errors per 100 | +0.0 (-2.7 to +2.7) | -0.7 (-2.7 to +2.0) | +4.0 (+0.7 to +8.0) clear | +2.7 (-0.7 to +6.0) | +2.0 (-1.3 to +5.3) |
| Error-free, points | +17.3 (+7.3 to +26.7) clear | -4.0 (-14.7 to +6.0) | -21.3 (-32.7 to -10.0) clear | -7.3 (-18.0 to +4.0) | -10.0 (-18.7 to +0.0) |
| Fluency | +0.25 (+0.11 to +0.38) clear | -0.27 (-0.44 to -0.09) clear | -0.37 (-0.53 to -0.21) clear | -0.03 (-0.16 to +0.11) | -0.11 (-0.23 to +0.01) |
| Key items wrong per 100 | -5.3 (-10.7 to +0.0) | +0.0 (-6.7 to +6.7) | +14.7 (+6.7 to +22.7) clear | +12.0 (+3.3 to +20.7) clear | +6.0 (-1.3 to +13.3) |
| Composite | +2.53 (+0.81 to +4.21) clear | -1.65 (-3.82 to +0.59) | -6.95 (-10.13 to -4.02) clear | -3.56 (-6.53 to -0.84) clear | -2.53 (-4.54 to -0.50) clear |

### Latin American Spanish

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 2.0 | 0.0 | 1.3 | 4.7 | 8.0 | 2.0 |
| Translations with a serious error, % | 2.0 | 0.0 | 1.3 | 4.7 | 7.3 | 2.0 |
| Critical errors per 100 | 0.7 | 0.0 | 0.0 | 0.7 | 1.3 | 1.3 |
| Major errors per 100 | 1.3 | 0.0 | 1.3 | 4.0 | 6.7 | 0.7 |
| Minor errors per 100 | 4.0 | 5.3 | 7.3 | 8.7 | 10.0 | 9.3 |
| Cosmetic errors per 100 | 6.7 | 6.0 | 15.3 | 21.3 | 18.0 | 22.0 |
| All errors per 100 | 14.7 | 12.7 | 39.3 | 40.0 | 39.3 | 37.3 |
| Error-free translations, % | 85.3 | 88.7 | 63.3 | 64.0 | 66.0 | 65.3 |
| Fluency, mean of 1 to 5 | 4.61 | 4.75 | 4.46 | 4.57 | 4.65 | 4.42 |
| Fluency 5, % | 68.0 | 80.0 | 56.0 | 62.7 | 69.3 | 50.0 |
| Fluency 4, % | 25.3 | 16.0 | 34.0 | 32.0 | 27.3 | 42.0 |
| Fluency 3, % | 6.7 | 3.3 | 10.0 | 5.3 | 2.0 | 8.0 |
| Fluency 2 or 1, % | 0.0 | 0.7 | 0.0 | 0.0 | 1.3 | 0.0 |
| Key items wrong per 100 | 2.0 | 0.0 | 1.3 | 6.0 | 7.3 | 2.7 |
| Translations with a key item wrong, % | 1.3 | 0.0 | 1.3 | 4.7 | 6.7 | 2.7 |
| Share of key items wrong, % | 0.53 | 0.00 | 0.35 | 1.58 | 1.93 | 0.70 |
| Unusable, count | 0 | 1 | 0 | 0 | 0 | 0 |
| Unusable, % | 0.00 | 0.67 | 0.00 | 0.00 | 0.00 | 0.00 |
| Composite, mean | 97.44 | 97.97 | 96.59 | 96.10 | 95.69 | 95.81 |
| Composite, median | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 95.86 |
| Composite, lowest 10% cut-off | 94.00 | 95.00 | 90.00 | 92.06 | 89.81 | 92.33 |
| Composite, double serious penalty | 97.08 | 97.97 | 96.32 | 95.14 | 94.06 | 95.14 |
| Serious fidelity errors per 100 | 1.3 | 0.0 | 1.3 | 3.3 | 6.7 | 0.7 |
| Serious logic errors per 100 | 0.0 | 0.0 | 0.0 | 0.0 | 0.7 | 0.7 |
| Serious numbers errors per 100 | 0.7 | 0.0 | 0.0 | 0.7 | 0.0 | 0.0 |
| Serious comprehension errors per 100 | 0.0 | 0.0 | 0.0 | 0.7 | 0.7 | 0.7 |
| All fidelity errors per 100 | 5.3 | 6.0 | 8.0 | 19.3 | 22.7 | 6.7 |
| All logic errors per 100 | 0.0 | 0.7 | 0.0 | 0.7 | 4.0 | 1.3 |
| All numbers errors per 100 | 0.7 | 0.0 | 0.0 | 2.0 | 0.7 | 2.7 |
| All comprehension errors per 100 | 6.7 | 4.7 | 16.0 | 12.7 | 8.7 | 22.7 |
| All form errors per 100 | 2.0 | 1.3 | 15.3 | 5.3 | 3.3 | 4.0 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -2.0 (-4.7 to +0.0) | -0.7 (-4.0 to +2.0) | +2.7 (-1.3 to +7.3) | +6.0 (+0.7 to +11.3) clear | +0.0 (-3.3 to +3.3) |
| Critical errors per 100 | -0.7 (-2.0 to +0.0) | -0.7 (-2.0 to +0.0) | +0.0 (-2.0 to +2.0) | +0.7 (-1.3 to +3.3) | +0.7 (-1.3 to +3.3) |
| Error-free, points | +3.3 (-4.7 to +10.7) | -22.0 (-32.7 to -12.0) clear | -21.3 (-30.7 to -13.3) clear | -19.3 (-28.7 to -10.0) clear | -20.0 (-30.0 to -11.3) clear |
| Fluency | +0.13 (+0.01 to +0.26) clear | -0.15 (-0.29 to -0.02) clear | -0.04 (-0.19 to +0.09) | +0.03 (-0.11 to +0.18) | -0.19 (-0.31 to -0.07) clear |
| Key items wrong per 100 | -2.0 (-5.3 to +0.0) | -0.7 (-4.7 to +2.7) | +4.0 (-1.3 to +9.3) | +5.3 (+0.0 to +10.7) | +0.7 (-4.0 to +4.7) |
| Composite | +0.53 (-1.13 to +1.83) | -0.85 (-1.94 to +0.31) | -1.34 (-2.95 to +0.15) | -1.75 (-3.51 to -0.07) clear | -1.64 (-3.09 to -0.15) clear |

### Persian

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 6.7 | 6.7 | 3.3 | 18.0 | 14.0 | 17.3 |
| Translations with a serious error, % | 6.7 | 6.7 | 3.3 | 16.0 | 12.7 | 14.7 |
| Critical errors per 100 | 2.0 | 3.3 | 0.0 | 8.0 | 3.3 | 2.7 |
| Major errors per 100 | 4.7 | 3.3 | 3.3 | 10.0 | 10.7 | 14.7 |
| Minor errors per 100 | 11.3 | 9.3 | 9.3 | 16.0 | 18.0 | 22.7 |
| Cosmetic errors per 100 | 15.3 | 12.7 | 14.7 | 24.0 | 21.3 | 16.0 |
| All errors per 100 | 36.0 | 32.7 | 37.3 | 61.3 | 54.7 | 58.0 |
| Error-free translations, % | 68.0 | 71.3 | 65.3 | 50.0 | 54.7 | 52.0 |
| Fluency, mean of 1 to 5 | 4.09 | 4.44 | 4.19 | 4.25 | 4.39 | 3.95 |
| Fluency 5, % | 22.7 | 55.3 | 29.3 | 42.0 | 47.3 | 18.7 |
| Fluency 4, % | 64.0 | 34.0 | 61.3 | 44.0 | 44.0 | 58.7 |
| Fluency 3, % | 13.3 | 10.0 | 8.0 | 11.3 | 8.7 | 22.0 |
| Fluency 2 or 1, % | 0.0 | 0.7 | 1.3 | 2.7 | 0.0 | 0.7 |
| Key items wrong per 100 | 6.0 | 8.0 | 3.3 | 17.3 | 15.3 | 18.0 |
| Translations with a key item wrong, % | 6.0 | 8.0 | 3.3 | 14.7 | 14.0 | 14.0 |
| Share of key items wrong, % | 1.58 | 2.11 | 0.88 | 4.57 | 4.04 | 4.75 |
| Unusable, count | 0 | 1 | 0 | 1 | 0 | 0 |
| Unusable, % | 0.00 | 0.67 | 0.00 | 0.67 | 0.00 | 0.00 |
| Composite, mean | 93.10 | 94.47 | 94.61 | 89.85 | 92.39 | 89.76 |
| Composite, median | 95.00 | 99.70 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 87.22 | 87.06 | 90.00 | 68.22 | 81.30 | 70.00 |
| Composite, double serious penalty | 91.55 | 93.04 | 93.99 | 86.24 | 89.72 | 86.53 |
| Serious fidelity errors per 100 | 2.7 | 1.3 | 1.3 | 6.0 | 8.7 | 9.3 |
| Serious logic errors per 100 | 1.3 | 1.3 | 0.0 | 1.3 | 2.0 | 2.0 |
| Serious numbers errors per 100 | 1.3 | 2.0 | 1.3 | 7.3 | 2.7 | 3.3 |
| Serious comprehension errors per 100 | 1.3 | 2.0 | 0.7 | 3.3 | 0.7 | 2.7 |
| All fidelity errors per 100 | 9.3 | 10.0 | 6.7 | 28.7 | 31.3 | 23.3 |
| All logic errors per 100 | 2.0 | 2.7 | 2.0 | 5.3 | 6.0 | 4.7 |
| All numbers errors per 100 | 3.3 | 3.3 | 7.3 | 8.7 | 4.7 | 5.3 |
| All comprehension errors per 100 | 18.7 | 12.7 | 11.3 | 15.3 | 11.3 | 22.7 |
| All form errors per 100 | 2.7 | 4.0 | 10.0 | 3.3 | 1.3 | 2.0 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | +0.0 (-5.3 to +6.7) | -3.3 (-8.7 to +2.0) | +11.3 (+3.3 to +19.3) clear | +7.3 (+0.7 to +15.3) clear | +10.7 (+4.0 to +18.7) clear |
| Critical errors per 100 | +1.3 (-2.0 to +5.3) | -2.0 (-4.7 to +0.0) | +6.0 (+1.3 to +11.3) clear | +1.3 (-2.0 to +5.3) | +0.7 (-2.7 to +4.0) |
| Error-free, points | +3.3 (-6.7 to +13.3) | -2.7 (-15.3 to +8.7) | -18.0 (-28.7 to -7.3) clear | -13.3 (-24.0 to -3.3) clear | -16.0 (-26.0 to -6.0) clear |
| Fluency | +0.35 (+0.21 to +0.49) clear | +0.09 (-0.06 to +0.23) | +0.16 (+0.03 to +0.31) clear | +0.29 (+0.16 to +0.43) clear | -0.14 (-0.27 to -0.01) clear |
| Key items wrong per 100 | +2.0 (-3.3 to +8.7) | -2.7 (-8.0 to +2.0) | +11.3 (+3.3 to +20.0) clear | +9.3 (+3.3 to +16.7) clear | +12.0 (+6.0 to +20.7) clear |
| Composite | +1.36 (-0.98 to +3.41) | +1.51 (-0.37 to +3.42) | -3.26 (-6.50 to -0.35) clear | -0.71 (-3.30 to +1.62) | -3.34 (-5.93 to -1.27) clear |

### Polish

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 7.3 | 1.3 | 6.7 | 10.0 | 16.7 | 17.3 |
| Translations with a serious error, % | 6.7 | 1.3 | 6.0 | 10.0 | 16.7 | 16.0 |
| Critical errors per 100 | 1.3 | 0.7 | 0.0 | 1.3 | 3.3 | 4.0 |
| Major errors per 100 | 6.0 | 0.7 | 6.7 | 8.7 | 13.3 | 13.3 |
| Minor errors per 100 | 24.0 | 9.3 | 17.3 | 26.7 | 26.0 | 31.3 |
| Cosmetic errors per 100 | 31.3 | 24.7 | 24.7 | 38.0 | 28.7 | 36.7 |
| All errors per 100 | 62.7 | 41.3 | 66.0 | 78.0 | 77.3 | 86.7 |
| Error-free translations, % | 48.0 | 63.3 | 47.3 | 42.0 | 38.7 | 36.0 |
| Fluency, mean of 1 to 5 | 4.00 | 4.31 | 3.95 | 4.01 | 4.09 | 3.82 |
| Fluency 5, % | 27.3 | 50.7 | 29.3 | 32.7 | 36.7 | 18.0 |
| Fluency 4, % | 47.3 | 31.3 | 38.7 | 38.7 | 40.0 | 49.3 |
| Fluency 3, % | 23.3 | 16.0 | 29.3 | 26.0 | 19.3 | 29.3 |
| Fluency 2 or 1, % | 2.0 | 2.0 | 2.7 | 2.7 | 4.0 | 3.3 |
| Key items wrong per 100 | 5.3 | 1.3 | 6.0 | 10.7 | 20.0 | 16.0 |
| Translations with a key item wrong, % | 5.3 | 1.3 | 5.3 | 9.3 | 18.0 | 13.3 |
| Share of key items wrong, % | 1.41 | 0.35 | 1.58 | 2.81 | 5.27 | 4.22 |
| Unusable, count | 0 | 0 | 0 | 0 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Composite, mean | 92.25 | 95.64 | 92.40 | 91.39 | 89.89 | 88.53 |
| Composite, median | 95.00 | 97.35 | 95.00 | 94.32 | 94.34 | 94.00 |
| Composite, lowest 10% cut-off | 85.00 | 89.00 | 86.00 | 81.00 | 73.01 | 69.56 |
| Composite, double serious penalty | 90.83 | 95.33 | 91.19 | 89.44 | 86.41 | 85.08 |
| Serious fidelity errors per 100 | 6.0 | 1.3 | 4.7 | 5.3 | 10.7 | 11.3 |
| Serious logic errors per 100 | 0.0 | 0.0 | 0.0 | 1.3 | 1.3 | 2.0 |
| Serious numbers errors per 100 | 0.7 | 0.0 | 0.7 | 2.0 | 2.7 | 2.7 |
| Serious comprehension errors per 100 | 0.7 | 0.0 | 1.3 | 1.3 | 2.0 | 1.3 |
| All fidelity errors per 100 | 25.3 | 12.0 | 8.7 | 24.7 | 28.7 | 29.3 |
| All logic errors per 100 | 0.7 | 2.0 | 1.3 | 5.3 | 2.7 | 2.7 |
| All numbers errors per 100 | 2.0 | 2.0 | 2.0 | 4.7 | 8.0 | 6.0 |
| All comprehension errors per 100 | 34.7 | 19.3 | 36.7 | 40.0 | 32.0 | 47.3 |
| All form errors per 100 | 0.0 | 6.0 | 17.3 | 3.3 | 6.0 | 1.3 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -6.0 (-11.3 to -1.3) clear | -0.7 (-7.3 to +6.0) | +2.7 (-3.3 to +10.0) | +9.3 (+2.0 to +16.0) clear | +10.0 (+2.7 to +17.3) clear |
| Critical errors per 100 | -0.7 (-3.3 to +1.3) | -1.3 (-3.3 to +0.0) | +0.0 (-2.7 to +2.7) | +2.0 (-0.7 to +4.7) | +2.7 (-0.7 to +6.7) |
| Error-free, points | +15.3 (+4.7 to +26.7) clear | -0.7 (-11.3 to +9.3) | -6.0 (-16.7 to +5.3) | -9.3 (-20.0 to +2.7) | -12.0 (-22.0 to -2.0) clear |
| Fluency | +0.31 (+0.14 to +0.48) clear | -0.05 (-0.23 to +0.13) | +0.01 (-0.13 to +0.19) | +0.09 (-0.09 to +0.26) | -0.18 (-0.35 to -0.02) clear |
| Key items wrong per 100 | -4.0 (-8.0 to +0.0) | +0.7 (-4.0 to +6.0) | +5.3 (-0.7 to +12.7) | +14.7 (+7.3 to +22.0) clear | +10.7 (+4.0 to +18.7) clear |
| Composite | +3.40 (+1.80 to +5.11) clear | +0.15 (-1.81 to +2.01) | -0.86 (-3.13 to +1.31) | -2.36 (-4.48 to -0.22) clear | -3.72 (-5.83 to -1.59) clear |

### Russian

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 6.7 | 6.0 | 3.3 | 5.3 | 15.3 | 11.3 |
| Translations with a serious error, % | 6.0 | 6.0 | 3.3 | 5.3 | 15.3 | 11.3 |
| Critical errors per 100 | 0.7 | 2.0 | 0.7 | 0.0 | 2.0 | 2.0 |
| Major errors per 100 | 6.0 | 4.0 | 2.7 | 5.3 | 13.3 | 9.3 |
| Minor errors per 100 | 10.0 | 4.7 | 10.7 | 17.3 | 14.7 | 26.7 |
| Cosmetic errors per 100 | 18.7 | 18.7 | 11.3 | 25.3 | 24.0 | 24.0 |
| All errors per 100 | 36.0 | 30.7 | 33.3 | 50.7 | 54.0 | 63.3 |
| Error-free translations, % | 70.7 | 71.3 | 70.7 | 56.0 | 54.7 | 46.7 |
| Fluency, mean of 1 to 5 | 4.30 | 4.63 | 4.27 | 4.51 | 4.49 | 3.95 |
| Fluency 5, % | 43.3 | 68.0 | 38.7 | 61.3 | 60.7 | 26.7 |
| Fluency 4, % | 44.7 | 27.3 | 50.0 | 30.7 | 28.7 | 44.0 |
| Fluency 3, % | 10.7 | 4.7 | 11.3 | 5.3 | 10.0 | 26.7 |
| Fluency 2 or 1, % | 1.3 | 0.0 | 0.0 | 2.7 | 0.7 | 2.7 |
| Key items wrong per 100 | 6.7 | 6.0 | 3.3 | 6.7 | 16.0 | 11.3 |
| Translations with a key item wrong, % | 6.0 | 6.0 | 3.3 | 6.7 | 16.0 | 10.7 |
| Share of key items wrong, % | 1.76 | 1.58 | 0.88 | 1.76 | 4.22 | 2.99 |
| Unusable, count | 0 | 0 | 0 | 1 | 0 | 1 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.67 | 0.00 | 0.67 |
| Composite, mean | 94.56 | 96.18 | 94.91 | 94.97 | 92.88 | 90.43 |
| Composite, median | 95.00 | 100.00 | 95.00 | 99.00 | 99.00 | 94.35 |
| Composite, lowest 10% cut-off | 86.38 | 92.26 | 89.37 | 87.33 | 75.00 | 79.74 |
| Composite, double serious penalty | 93.42 | 94.76 | 94.11 | 94.07 | 89.94 | 88.37 |
| Serious fidelity errors per 100 | 4.0 | 4.0 | 1.3 | 4.0 | 9.3 | 6.0 |
| Serious logic errors per 100 | 0.7 | 1.3 | 0.7 | 0.7 | 2.7 | 2.7 |
| Serious numbers errors per 100 | 1.3 | 0.7 | 1.3 | 0.7 | 3.3 | 2.7 |
| Serious comprehension errors per 100 | 0.7 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| All fidelity errors per 100 | 12.0 | 12.7 | 4.7 | 29.3 | 26.0 | 16.0 |
| All logic errors per 100 | 2.0 | 4.0 | 2.7 | 2.7 | 8.0 | 6.0 |
| All numbers errors per 100 | 3.3 | 2.0 | 3.3 | 4.0 | 4.7 | 4.7 |
| All comprehension errors per 100 | 18.0 | 10.7 | 14.7 | 12.0 | 15.3 | 35.3 |
| All form errors per 100 | 0.7 | 1.3 | 8.0 | 2.7 | 0.0 | 1.3 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -0.7 (-6.7 to +5.3) | -3.3 (-8.7 to +1.3) | -1.3 (-8.0 to +4.7) | +8.7 (+2.0 to +15.3) clear | +4.7 (-0.7 to +10.0) |
| Critical errors per 100 | +1.3 (-1.3 to +4.0) | +0.0 (-2.0 to +2.0) | -0.7 (-2.0 to +0.0) | +1.3 (-1.3 to +4.0) | +1.3 (-0.7 to +4.0) |
| Error-free, points | +0.7 (-8.0 to +9.3) | +0.0 (-9.3 to +8.7) | -14.7 (-24.7 to -4.0) clear | -16.0 (-26.0 to -4.7) clear | -24.0 (-33.3 to -14.0) clear |
| Fluency | +0.33 (+0.20 to +0.46) clear | -0.03 (-0.16 to +0.09) | +0.21 (+0.05 to +0.36) clear | +0.19 (+0.04 to +0.35) clear | -0.35 (-0.50 to -0.22) clear |
| Key items wrong per 100 | -0.7 (-6.7 to +5.3) | -3.3 (-9.3 to +2.0) | +0.0 (-6.7 to +6.0) | +9.3 (+2.7 to +16.0) clear | +4.7 (-0.7 to +10.0) |
| Composite | +1.62 (-0.21 to +3.36) | +0.36 (-1.39 to +2.01) | +0.41 (-1.80 to +2.54) | -1.68 (-3.78 to +0.14) | -4.13 (-6.05 to -2.33) clear |

### Simplified Chinese

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 2.7 | 2.7 | 4.7 | 6.7 | 13.3 | 24.0 |
| Translations with a serious error, % | 2.7 | 2.7 | 3.3 | 6.7 | 12.7 | 20.0 |
| Critical errors per 100 | 0.0 | 0.7 | 0.7 | 2.7 | 2.7 | 11.3 |
| Major errors per 100 | 2.7 | 2.0 | 4.0 | 4.0 | 10.7 | 12.7 |
| Minor errors per 100 | 6.7 | 2.0 | 16.7 | 8.7 | 14.7 | 18.0 |
| Cosmetic errors per 100 | 13.3 | 6.0 | 16.0 | 17.3 | 16.7 | 20.0 |
| All errors per 100 | 24.0 | 11.3 | 52.7 | 33.3 | 46.7 | 62.0 |
| Error-free translations, % | 78.7 | 88.7 | 58.7 | 68.0 | 59.3 | 48.0 |
| Fluency, mean of 1 to 5 | 4.45 | 4.71 | 4.06 | 4.69 | 4.66 | 4.13 |
| Fluency 5, % | 50.0 | 72.0 | 30.7 | 70.0 | 66.7 | 31.3 |
| Fluency 4, % | 45.3 | 26.7 | 46.7 | 29.3 | 32.7 | 50.7 |
| Fluency 3, % | 4.7 | 1.3 | 20.7 | 0.7 | 0.7 | 17.3 |
| Fluency 2 or 1, % | 0.0 | 0.0 | 2.0 | 0.0 | 0.0 | 0.7 |
| Key items wrong per 100 | 2.7 | 2.7 | 4.7 | 7.3 | 12.0 | 22.0 |
| Translations with a key item wrong, % | 2.7 | 2.7 | 3.3 | 5.3 | 12.0 | 19.3 |
| Share of key items wrong, % | 0.70 | 0.70 | 1.23 | 1.93 | 3.16 | 5.80 |
| Unusable, count | 0 | 0 | 0 | 0 | 1 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.67 | 0.00 |
| Composite, mean | 96.36 | 97.67 | 93.56 | 95.81 | 93.92 | 87.32 |
| Composite, median | 95.00 | 100.00 | 95.00 | 100.00 | 99.36 | 95.00 |
| Composite, lowest 10% cut-off | 94.00 | 95.00 | 86.67 | 94.00 | 77.30 | 60.03 |
| Composite, double serious penalty | 95.92 | 97.08 | 92.73 | 94.09 | 91.40 | 82.21 |
| Serious fidelity errors per 100 | 1.3 | 0.7 | 2.7 | 4.7 | 6.0 | 14.7 |
| Serious logic errors per 100 | 0.7 | 0.0 | 0.7 | 0.7 | 3.3 | 2.7 |
| Serious numbers errors per 100 | 0.7 | 2.0 | 1.3 | 1.3 | 4.0 | 5.3 |
| Serious comprehension errors per 100 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.3 |
| All fidelity errors per 100 | 8.0 | 2.7 | 10.0 | 20.7 | 24.0 | 28.7 |
| All logic errors per 100 | 2.0 | 2.7 | 2.0 | 4.0 | 8.0 | 4.7 |
| All numbers errors per 100 | 1.3 | 2.0 | 6.0 | 6.0 | 9.3 | 10.0 |
| All comprehension errors per 100 | 11.3 | 3.3 | 19.3 | 2.0 | 3.3 | 18.7 |
| All form errors per 100 | 1.3 | 0.7 | 15.3 | 0.7 | 2.0 | 0.0 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | +0.0 (-4.0 to +3.3) | +2.0 (-2.7 to +8.0) | +4.0 (-0.7 to +8.7) | +10.7 (+4.7 to +17.3) clear | +21.3 (+12.0 to +29.3) clear |
| Critical errors per 100 | +0.7 (+0.0 to +2.0) | +0.7 (+0.0 to +2.0) | +2.7 (+0.7 to +5.3) clear | +2.7 (+0.7 to +5.3) clear | +11.3 (+6.0 to +16.7) clear |
| Error-free, points | +10.0 (+1.3 to +18.7) clear | -20.0 (-30.0 to -9.3) clear | -10.7 (-20.7 to -1.3) clear | -19.3 (-29.3 to -8.0) clear | -30.7 (-40.7 to -20.7) clear |
| Fluency | +0.25 (+0.14 to +0.37) clear | -0.39 (-0.53 to -0.25) clear | +0.24 (+0.13 to +0.36) clear | +0.21 (+0.09 to +0.33) clear | -0.33 (-0.46 to -0.20) clear |
| Key items wrong per 100 | +0.0 (-4.0 to +3.3) | +2.0 (-2.7 to +7.3) | +4.7 (-0.7 to +10.7) | +9.3 (+4.0 to +15.3) clear | +19.3 (+11.3 to +27.3) clear |
| Composite | +1.30 (+0.23 to +2.39) clear | -2.81 (-4.36 to -1.44) clear | -0.55 (-2.27 to +1.19) | -2.44 (-4.90 to -0.46) clear | -9.05 (-11.89 to -5.94) clear |

### Thai

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 6.0 | 4.0 | 2.0 | 17.3 | 14.7 | 14.0 |
| Translations with a serious error, % | 5.3 | 4.0 | 2.0 | 16.0 | 13.3 | 14.0 |
| Critical errors per 100 | 0.7 | 0.7 | 0.0 | 4.7 | 4.7 | 3.3 |
| Major errors per 100 | 5.3 | 3.3 | 2.0 | 12.7 | 10.0 | 10.7 |
| Minor errors per 100 | 12.7 | 2.7 | 4.7 | 20.0 | 20.0 | 15.3 |
| Cosmetic errors per 100 | 14.0 | 10.7 | 8.7 | 32.7 | 27.3 | 18.7 |
| All errors per 100 | 34.0 | 19.3 | 18.0 | 74.7 | 63.3 | 48.0 |
| Error-free translations, % | 71.3 | 81.3 | 84.0 | 44.0 | 52.0 | 58.0 |
| Fluency, mean of 1 to 5 | 4.25 | 4.52 | 4.43 | 4.19 | 4.30 | 4.25 |
| Fluency 5, % | 34.0 | 56.0 | 46.0 | 35.3 | 38.0 | 34.7 |
| Fluency 4, % | 58.0 | 40.0 | 50.7 | 50.0 | 54.0 | 55.3 |
| Fluency 3, % | 6.7 | 4.0 | 3.3 | 12.7 | 8.0 | 10.0 |
| Fluency 2 or 1, % | 1.3 | 0.0 | 0.0 | 2.0 | 0.0 | 0.0 |
| Key items wrong per 100 | 7.3 | 3.3 | 2.0 | 19.3 | 16.0 | 18.0 |
| Translations with a key item wrong, % | 6.7 | 2.7 | 2.0 | 17.3 | 13.3 | 16.0 |
| Share of key items wrong, % | 1.93 | 0.88 | 0.53 | 5.10 | 4.22 | 4.75 |
| Unusable, count | 1 | 0 | 0 | 3 | 1 | 0 |
| Unusable, % | 0.67 | 0.00 | 0.00 | 2.00 | 0.67 | 0.00 |
| Composite, mean | 93.91 | 96.51 | 96.38 | 88.20 | 91.02 | 91.59 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 87.75 | 94.00 | 94.32 | 68.33 | 69.00 | 76.00 |
| Composite, double serious penalty | 93.01 | 95.76 | 95.98 | 84.44 | 87.84 | 88.61 |
| Serious fidelity errors per 100 | 4.0 | 2.7 | 0.7 | 9.3 | 8.0 | 7.3 |
| Serious logic errors per 100 | 1.3 | 1.3 | 0.7 | 2.7 | 2.0 | 3.3 |
| Serious numbers errors per 100 | 0.7 | 0.0 | 0.7 | 4.0 | 3.3 | 3.3 |
| Serious comprehension errors per 100 | 0.0 | 0.0 | 0.0 | 1.3 | 1.3 | 0.0 |
| All fidelity errors per 100 | 20.7 | 7.3 | 6.7 | 40.0 | 33.3 | 26.0 |
| All logic errors per 100 | 3.3 | 2.0 | 1.3 | 5.3 | 6.0 | 3.3 |
| All numbers errors per 100 | 2.7 | 4.7 | 1.3 | 12.0 | 9.3 | 8.0 |
| All comprehension errors per 100 | 6.0 | 3.3 | 6.0 | 12.7 | 13.3 | 10.7 |
| All form errors per 100 | 1.3 | 2.0 | 2.7 | 4.7 | 1.3 | 0.0 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -2.0 (-8.0 to +2.7) | -4.0 (-9.3 to +0.7) | +11.3 (+3.3 to +20.0) clear | +8.7 (+1.3 to +16.7) clear | +8.0 (+2.0 to +14.0) clear |
| Critical errors per 100 | +0.0 (-2.0 to +2.0) | -0.7 (-2.0 to +0.0) | +4.0 (+0.7 to +8.0) clear | +4.0 (+0.7 to +8.0) clear | +2.7 (-0.7 to +6.0) |
| Error-free, points | +10.0 (+0.0 to +19.3) | +12.7 (+4.0 to +22.0) clear | -27.3 (-37.3 to -16.7) clear | -19.3 (-29.3 to -8.7) clear | -13.3 (-23.3 to -4.0) clear |
| Fluency | +0.27 (+0.15 to +0.39) clear | +0.18 (+0.05 to +0.31) clear | -0.06 (-0.21 to +0.09) | +0.05 (-0.06 to +0.17) | +0.00 (-0.13 to +0.13) |
| Key items wrong per 100 | -4.0 (-10.0 to +1.3) | -5.3 (-11.3 to -0.7) clear | +12.0 (+2.7 to +20.0) clear | +8.7 (+0.0 to +16.0) | +10.7 (+3.3 to +17.3) clear |
| Composite | +2.60 (+0.85 to +4.65) clear | +2.47 (+0.71 to +4.81) clear | -5.71 (-9.47 to -2.13) clear | -2.89 (-5.42 to +0.17) | -2.32 (-4.64 to +0.19) |

### Traditional Chinese

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 4.0 | 3.3 | 6.0 | 7.3 | 16.7 | 22.0 |
| Translations with a serious error, % | 4.0 | 3.3 | 6.0 | 7.3 | 15.3 | 21.3 |
| Critical errors per 100 | 0.7 | 0.7 | 0.0 | 0.7 | 2.7 | 8.7 |
| Major errors per 100 | 3.3 | 2.7 | 6.0 | 6.7 | 14.0 | 13.3 |
| Minor errors per 100 | 5.3 | 4.0 | 16.7 | 7.3 | 16.7 | 23.3 |
| Cosmetic errors per 100 | 10.0 | 5.3 | 17.3 | 20.0 | 25.3 | 19.3 |
| All errors per 100 | 20.0 | 14.0 | 52.7 | 54.0 | 61.3 | 65.3 |
| Error-free translations, % | 81.3 | 86.0 | 54.0 | 50.7 | 52.7 | 50.0 |
| Fluency, mean of 1 to 5 | 4.40 | 4.79 | 3.90 | 4.19 | 4.41 | 4.07 |
| Fluency 5, % | 45.3 | 80.7 | 21.3 | 38.0 | 45.3 | 27.3 |
| Fluency 4, % | 50.0 | 17.3 | 48.0 | 45.3 | 50.7 | 54.7 |
| Fluency 3, % | 4.0 | 2.0 | 30.0 | 14.7 | 4.0 | 16.0 |
| Fluency 2 or 1, % | 0.7 | 0.0 | 0.7 | 2.0 | 0.0 | 2.0 |
| Key items wrong per 100 | 4.7 | 4.0 | 6.7 | 6.7 | 16.0 | 24.0 |
| Translations with a key item wrong, % | 4.0 | 4.0 | 6.7 | 6.7 | 15.3 | 22.0 |
| Share of key items wrong, % | 1.23 | 1.05 | 1.76 | 1.76 | 4.22 | 6.33 |
| Unusable, count | 0 | 1 | 0 | 11 | 0 | 2 |
| Unusable, % | 0.00 | 0.67 | 0.00 | 7.33 | 0.00 | 1.33 |
| Composite, mean | 95.80 | 97.32 | 92.50 | 87.29 | 92.21 | 87.06 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 94.00 | 95.00 | 86.00 | 72.33 | 74.78 | 63.33 |
| Composite, double serious penalty | 95.09 | 96.72 | 91.51 | 86.03 | 88.97 | 82.18 |
| Serious fidelity errors per 100 | 2.0 | 1.3 | 2.0 | 4.0 | 8.7 | 14.0 |
| Serious logic errors per 100 | 1.3 | 0.0 | 1.3 | 2.0 | 2.7 | 2.7 |
| Serious numbers errors per 100 | 0.0 | 2.0 | 2.7 | 0.7 | 4.0 | 4.0 |
| Serious comprehension errors per 100 | 0.7 | 0.0 | 0.0 | 0.7 | 1.3 | 1.3 |
| All fidelity errors per 100 | 5.3 | 6.0 | 10.0 | 18.7 | 32.0 | 30.7 |
| All logic errors per 100 | 4.0 | 0.7 | 1.3 | 4.7 | 8.7 | 7.3 |
| All numbers errors per 100 | 2.0 | 4.7 | 4.0 | 2.7 | 9.3 | 10.0 |
| All comprehension errors per 100 | 8.0 | 1.3 | 24.7 | 8.7 | 8.7 | 16.7 |
| All form errors per 100 | 0.7 | 1.3 | 12.7 | 19.3 | 2.7 | 0.7 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -0.7 (-4.7 to +3.3) | +2.0 (-2.7 to +6.7) | +3.3 (-1.3 to +8.0) | +12.7 (+6.0 to +20.0) clear | +18.0 (+11.3 to +24.7) clear |
| Critical errors per 100 | +0.0 (-2.0 to +2.0) | -0.7 (-2.0 to +0.0) | +0.0 (-2.0 to +2.0) | +2.0 (-0.7 to +4.7) | +8.0 (+3.3 to +12.7) clear |
| Error-free, points | +4.7 (-2.7 to +12.7) | -27.3 (-36.0 to -17.3) clear | -30.7 (-39.3 to -20.7) clear | -28.7 (-38.7 to -19.3) clear | -31.3 (-41.3 to -23.3) clear |
| Fluency | +0.39 (+0.28 to +0.50) clear | -0.50 (-0.63 to -0.35) clear | -0.21 (-0.35 to -0.06) clear | +0.01 (-0.11 to +0.13) | -0.33 (-0.46 to -0.19) clear |
| Key items wrong per 100 | -0.7 (-6.0 to +4.0) | +2.0 (-2.7 to +7.3) | +2.0 (-3.3 to +6.7) | +11.3 (+4.7 to +18.7) clear | +19.3 (+11.3 to +27.3) clear |
| Composite | +1.52 (-0.46 to +3.21) | -3.30 (-4.67 to -1.77) clear | -8.51 (-12.73 to -4.24) clear | -3.59 (-6.03 to -1.71) clear | -8.74 (-11.68 to -5.88) clear |

### Turkish

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 6.0 | 4.7 | 4.0 | 10.7 | 15.3 | 17.3 |
| Translations with a serious error, % | 6.0 | 4.7 | 4.0 | 9.3 | 14.7 | 16.0 |
| Critical errors per 100 | 0.0 | 0.0 | 0.0 | 2.7 | 3.3 | 5.3 |
| Major errors per 100 | 6.0 | 4.7 | 4.0 | 8.0 | 12.0 | 12.0 |
| Minor errors per 100 | 12.7 | 10.0 | 10.0 | 16.0 | 12.7 | 26.7 |
| Cosmetic errors per 100 | 21.3 | 21.3 | 13.3 | 17.3 | 15.3 | 18.7 |
| All errors per 100 | 40.7 | 38.0 | 42.0 | 46.7 | 44.7 | 65.3 |
| Error-free translations, % | 63.3 | 68.0 | 61.3 | 60.7 | 64.0 | 46.7 |
| Fluency, mean of 1 to 5 | 4.22 | 4.51 | 4.25 | 4.43 | 4.53 | 4.03 |
| Fluency 5, % | 36.7 | 63.3 | 34.0 | 52.0 | 60.7 | 26.0 |
| Fluency 4, % | 50.7 | 24.7 | 56.7 | 40.0 | 32.0 | 52.7 |
| Fluency 3, % | 10.7 | 12.0 | 9.3 | 7.3 | 7.3 | 20.0 |
| Fluency 2 or 1, % | 2.0 | 0.0 | 0.0 | 0.7 | 0.0 | 1.3 |
| Key items wrong per 100 | 6.0 | 4.7 | 4.7 | 10.0 | 18.0 | 23.3 |
| Translations with a key item wrong, % | 6.0 | 4.7 | 4.7 | 8.7 | 16.0 | 19.3 |
| Share of key items wrong, % | 1.58 | 1.23 | 1.23 | 2.64 | 4.75 | 6.15 |
| Unusable, count | 0 | 0 | 0 | 0 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Composite, mean | 94.16 | 95.99 | 94.84 | 93.91 | 92.87 | 89.21 |
| Composite, median | 95.00 | 100.00 | 95.00 | 95.00 | 99.00 | 94.67 |
| Composite, lowest 10% cut-off | 86.20 | 89.00 | 88.53 | 82.30 | 77.84 | 72.67 |
| Composite, double serious penalty | 93.13 | 95.18 | 94.12 | 92.06 | 89.70 | 85.52 |
| Serious fidelity errors per 100 | 2.7 | 2.0 | 2.7 | 5.3 | 4.7 | 6.0 |
| Serious logic errors per 100 | 0.7 | 0.7 | 0.0 | 1.3 | 4.0 | 4.0 |
| Serious numbers errors per 100 | 2.7 | 1.3 | 1.3 | 3.3 | 6.7 | 6.7 |
| Serious comprehension errors per 100 | 0.0 | 0.7 | 0.0 | 0.7 | 0.0 | 0.7 |
| All fidelity errors per 100 | 10.7 | 12.7 | 6.7 | 17.3 | 20.7 | 18.0 |
| All logic errors per 100 | 2.7 | 1.3 | 1.3 | 4.7 | 5.3 | 6.7 |
| All numbers errors per 100 | 4.0 | 4.7 | 4.7 | 6.7 | 8.7 | 9.3 |
| All comprehension errors per 100 | 22.7 | 17.3 | 14.7 | 15.3 | 8.7 | 28.7 |
| All form errors per 100 | 0.7 | 2.0 | 14.7 | 2.7 | 1.3 | 2.7 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -1.3 (-6.0 to +3.3) | -2.0 (-6.7 to +2.7) | +4.7 (-2.0 to +12.0) | +9.3 (+2.7 to +16.7) clear | +11.3 (+4.7 to +19.3) clear |
| Critical errors per 100 | +0.0 (+0.0 to +0.0) | +0.0 (+0.0 to +0.0) | +2.7 (+0.7 to +5.3) clear | +3.3 (+0.7 to +6.7) clear | +5.3 (+2.0 to +9.3) clear |
| Error-free, points | +4.7 (-4.7 to +14.7) | -2.0 (-12.7 to +8.0) | -2.7 (-12.7 to +8.0) | +0.7 (-10.0 to +11.3) | -16.7 (-26.0 to -7.3) clear |
| Fluency | +0.29 (+0.15 to +0.43) clear | +0.03 (-0.09 to +0.15) | +0.21 (+0.06 to +0.36) clear | +0.31 (+0.17 to +0.43) clear | -0.19 (-0.31 to -0.07) clear |
| Key items wrong per 100 | -1.3 (-6.0 to +3.3) | -1.3 (-6.0 to +3.3) | +4.0 (-2.7 to +10.7) | +12.0 (+5.3 to +20.7) clear | +17.3 (+9.3 to +26.0) clear |
| Composite | +1.83 (+0.16 to +3.27) clear | +0.68 (-0.78 to +1.92) | -0.25 (-2.49 to +1.63) | -1.30 (-3.76 to +0.67) | -4.95 (-7.18 to -3.05) clear |

### Ukrainian

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 6.0 | 3.3 | 5.3 | 13.3 | 17.3 | 10.7 |
| Translations with a serious error, % | 6.0 | 3.3 | 5.3 | 11.3 | 17.3 | 10.7 |
| Critical errors per 100 | 0.0 | 0.7 | 1.3 | 4.0 | 2.0 | 2.7 |
| Major errors per 100 | 6.0 | 2.7 | 4.0 | 9.3 | 15.3 | 8.0 |
| Minor errors per 100 | 12.7 | 14.7 | 11.3 | 34.7 | 10.0 | 28.7 |
| Cosmetic errors per 100 | 18.0 | 20.7 | 17.3 | 39.3 | 20.0 | 30.7 |
| All errors per 100 | 39.3 | 40.0 | 46.0 | 92.0 | 48.7 | 74.7 |
| Error-free translations, % | 64.7 | 65.3 | 58.7 | 34.7 | 56.0 | 44.0 |
| Fluency, mean of 1 to 5 | 4.23 | 4.37 | 4.20 | 3.97 | 4.47 | 3.83 |
| Fluency 5, % | 39.3 | 46.7 | 35.3 | 31.3 | 50.7 | 20.7 |
| Fluency 4, % | 46.7 | 44.7 | 51.3 | 40.7 | 46.7 | 46.7 |
| Fluency 3, % | 12.0 | 7.3 | 11.3 | 22.0 | 2.0 | 27.3 |
| Fluency 2 or 1, % | 2.0 | 1.3 | 2.0 | 6.0 | 0.7 | 5.3 |
| Key items wrong per 100 | 6.0 | 2.7 | 4.7 | 10.7 | 17.3 | 12.0 |
| Translations with a key item wrong, % | 6.0 | 2.7 | 4.7 | 10.0 | 16.7 | 9.3 |
| Share of key items wrong, % | 1.58 | 0.70 | 1.23 | 2.81 | 4.57 | 3.16 |
| Unusable, count | 0 | 0 | 0 | 1 | 0 | 2 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 0.67 | 0.00 | 1.33 |
| Composite, mean | 94.28 | 95.49 | 93.99 | 89.37 | 92.48 | 89.06 |
| Composite, median | 95.00 | 95.00 | 95.00 | 94.28 | 95.00 | 94.28 |
| Composite, lowest 10% cut-off | 87.01 | 91.00 | 86.71 | 74.21 | 73.33 | 75.56 |
| Composite, double serious penalty | 93.24 | 94.88 | 92.77 | 86.93 | 89.03 | 87.00 |
| Serious fidelity errors per 100 | 2.0 | 2.7 | 2.7 | 7.3 | 9.3 | 6.7 |
| Serious logic errors per 100 | 0.7 | 0.0 | 1.3 | 2.0 | 3.3 | 2.7 |
| Serious numbers errors per 100 | 2.0 | 0.0 | 0.7 | 2.0 | 4.0 | 0.0 |
| Serious comprehension errors per 100 | 1.3 | 0.7 | 0.7 | 2.0 | 0.7 | 1.3 |
| All fidelity errors per 100 | 10.7 | 14.7 | 8.0 | 34.7 | 26.0 | 21.3 |
| All logic errors per 100 | 1.3 | 0.7 | 1.3 | 4.7 | 4.7 | 4.7 |
| All numbers errors per 100 | 2.0 | 3.3 | 2.7 | 4.0 | 6.0 | 2.0 |
| All comprehension errors per 100 | 22.7 | 20.0 | 22.0 | 44.0 | 10.7 | 42.0 |
| All form errors per 100 | 2.7 | 1.3 | 12.0 | 4.7 | 1.3 | 4.7 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | -2.7 (-7.3 to +1.3) | -0.7 (-6.0 to +5.3) | +7.3 (-0.7 to +14.7) | +11.3 (+4.7 to +18.7) clear | +4.7 (-0.7 to +10.7) |
| Critical errors per 100 | +0.7 (+0.0 to +2.0) | +1.3 (+0.0 to +3.3) | +4.0 (+0.7 to +8.0) clear | +2.0 (+0.0 to +4.7) | +2.7 (+0.7 to +5.3) clear |
| Error-free, points | +0.7 (-8.7 to +11.3) | -6.0 (-16.7 to +4.7) | -30.0 (-41.3 to -18.7) clear | -8.7 (-20.0 to +3.3) | -20.7 (-30.7 to -10.7) clear |
| Fluency | +0.13 (-0.02 to +0.27) | -0.03 (-0.21 to +0.13) | -0.26 (-0.45 to -0.07) clear | +0.24 (+0.09 to +0.37) clear | -0.41 (-0.57 to -0.25) clear |
| Key items wrong per 100 | -3.3 (-7.3 to +0.7) | -1.3 (-6.7 to +4.0) | +4.7 (-2.7 to +12.0) | +11.3 (+5.3 to +18.7) clear | +6.0 (-0.7 to +13.3) |
| Composite | +1.21 (-0.32 to +2.66) | -0.29 (-2.14 to +1.51) | -4.91 (-7.89 to -2.32) clear | -1.81 (-3.96 to +0.38) | -5.22 (-8.06 to -2.56) clear |

### Vietnamese

150 translations per system, from 150 source texts.

| Metric | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|---|
| Serious errors per 100 | 1.3 | 1.3 | 1.3 | 8.7 | 10.0 | 14.0 |
| Translations with a serious error, % | 1.3 | 1.3 | 1.3 | 8.0 | 10.0 | 12.7 |
| Critical errors per 100 | 0.0 | 1.3 | 0.0 | 2.7 | 0.7 | 2.7 |
| Major errors per 100 | 1.3 | 0.0 | 1.3 | 6.0 | 9.3 | 11.3 |
| Minor errors per 100 | 5.3 | 4.0 | 8.0 | 8.7 | 12.0 | 16.7 |
| Cosmetic errors per 100 | 6.7 | 9.3 | 15.3 | 20.7 | 24.7 | 16.7 |
| All errors per 100 | 14.0 | 15.3 | 36.7 | 39.3 | 48.7 | 48.7 |
| Error-free translations, % | 87.3 | 84.7 | 67.3 | 64.7 | 59.3 | 59.3 |
| Fluency, mean of 1 to 5 | 4.57 | 4.54 | 4.23 | 4.45 | 4.47 | 4.16 |
| Fluency 5, % | 58.0 | 56.0 | 32.7 | 51.3 | 50.0 | 30.0 |
| Fluency 4, % | 41.3 | 42.0 | 58.0 | 42.0 | 47.3 | 56.0 |
| Fluency 3, % | 0.7 | 2.0 | 9.3 | 6.7 | 2.7 | 14.0 |
| Fluency 2 or 1, % | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Key items wrong per 100 | 2.0 | 1.3 | 0.7 | 8.0 | 11.3 | 13.3 |
| Translations with a key item wrong, % | 2.0 | 1.3 | 0.7 | 7.3 | 10.0 | 12.0 |
| Share of key items wrong, % | 0.53 | 0.35 | 0.18 | 2.11 | 2.99 | 3.51 |
| Unusable, count | 0 | 0 | 0 | 2 | 0 | 0 |
| Unusable, % | 0.00 | 0.00 | 0.00 | 1.33 | 0.00 | 0.00 |
| Composite, mean | 97.37 | 96.98 | 95.50 | 93.51 | 94.13 | 91.37 |
| Composite, median | 100.00 | 100.00 | 95.00 | 95.00 | 95.00 | 95.00 |
| Composite, lowest 10% cut-off | 94.39 | 94.32 | 90.00 | 89.00 | 86.00 | 75.33 |
| Composite, double serious penalty | 97.19 | 96.53 | 95.23 | 92.13 | 92.19 | 88.65 |
| Serious fidelity errors per 100 | 1.3 | 1.3 | 1.3 | 4.7 | 5.3 | 8.0 |
| Serious logic errors per 100 | 0.0 | 0.0 | 0.0 | 1.3 | 2.0 | 4.0 |
| Serious numbers errors per 100 | 0.0 | 0.0 | 0.0 | 2.0 | 2.0 | 1.3 |
| Serious comprehension errors per 100 | 0.0 | 0.0 | 0.0 | 0.7 | 0.7 | 0.7 |
| All fidelity errors per 100 | 6.7 | 4.0 | 6.0 | 20.7 | 30.7 | 13.3 |
| All logic errors per 100 | 0.7 | 2.7 | 1.3 | 4.0 | 6.0 | 6.0 |
| All numbers errors per 100 | 1.3 | 1.3 | 3.3 | 2.0 | 4.7 | 10.0 |
| All comprehension errors per 100 | 4.7 | 6.7 | 14.0 | 11.3 | 5.3 | 18.0 |
| All form errors per 100 | 0.7 | 0.7 | 12.0 | 1.3 | 2.0 | 1.3 |

Each system minus OpenTalk, with a 95% interval that resamples whole source texts. "clear" means the interval does not cross 0.

| Measure | Haiku | 31B | Qwen | Tencent | Plain E4B |
|---|---|---|---|---|---|
| Serious errors per 100 | +0.0 (-2.7 to +2.7) | +0.0 (-2.7 to +2.7) | +7.3 (+2.0 to +12.7) clear | +8.7 (+4.0 to +13.3) clear | +12.7 (+7.3 to +18.7) clear |
| Critical errors per 100 | +1.3 (+0.0 to +3.3) | +0.0 (+0.0 to +0.0) | +2.7 (+0.7 to +5.3) clear | +0.7 (+0.0 to +2.0) | +2.7 (+0.7 to +5.3) clear |
| Error-free, points | -2.7 (-10.0 to +4.7) | -20.0 (-28.0 to -10.7) clear | -22.7 (-32.7 to -13.3) clear | -28.0 (-37.3 to -19.3) clear | -28.0 (-36.7 to -20.0) clear |
| Fluency | -0.03 (-0.14 to +0.06) | -0.34 (-0.45 to -0.23) clear | -0.13 (-0.27 to -0.01) clear | -0.10 (-0.20 to +0.01) | -0.41 (-0.53 to -0.29) clear |
| Key items wrong per 100 | -0.7 (-3.3 to +2.0) | -1.3 (-4.0 to +0.7) | +6.0 (+0.7 to +11.3) clear | +9.3 (+3.3 to +15.3) clear | +11.3 (+5.3 to +18.0) clear |
| Composite | -0.39 (-1.39 to +0.50) | -1.87 (-2.62 to -1.07) clear | -3.86 (-6.68 to -1.48) clear | -3.24 (-4.90 to -1.82) clear | -6.00 (-8.10 to -4.16) clear |

## Texts with the most serious errors, OpenTalk

| Text | Words | OpenTalk | Haiku | 31B | Qwen | Tencent | Plain E4B | Source |
|---|---|---|---|---|---|---|---|---|
| f084 | 37 | 10 | 2 | 1 | 6 | 7 | 4 | The scan shows the lesion is not unlikely to be residual tumor rather than scar, so we'll repeat imaging in si |
| g050 | 73 | 7 | 4 | 4 | 2 | 2 | 6 | Your daughter will get two shots today. She might have a sore arm and a low fever for a day or two. You can gi |
| f077 | 78 | 6 | 0 | 0 | 0 | 0 | 2 | Your thyroid nodule biopsy came back as indeterminate, which is a gray zone, so I can't tell you it's cancer o |
| f068 | 75 | 5 | 0 | 2 | 6 | 9 | 7 | Your dad fell last night, and the head scan is clear, which is great news. But he takes a blood thinner, so we |
| f017 | 19 | 4 | 1 | 2 | 4 | 17 | 21 | I'm going to hold your Eliquis the morning of the procedure, but don't skip it any other day, okay? |
| f052 | 70 | 4 | 2 | 0 | 2 | 0 | 10 | Your gallbladder is full of stones and it's inflamed, which explains the pain after fatty meals. The surgery i |
| f055 | 76 | 4 | 0 | 0 | 1 | 0 | 2 | Your sugars have been running high in the afternoons, mostly in the two hundreds, so I want to change the plan |
| f057 | 76 | 4 | 0 | 1 | 3 | 4 | 5 | For your asthma, the purple inhaler is your controller, so you use it every day, two puffs in the morning and  |
| f079 | 76 | 4 | 2 | 1 | 5 | 4 | 4 | The wrist X-ray shows a break, but the bones are lined up well, so we don't need to set it. We'll put on a spl |
| f082 | 36 | 4 | 0 | 4 | 7 | 3 | 11 | We can't rule out that the baby is breech, so unless your water breaks or contractions come closer than five m |
| f007 | 27 | 3 | 0 | 2 | 3 | 7 | 1 | We can't rule out a small fracture on the X-ray, so I'm putting you in a boot, and no weight on it until the o |
| f060 | 73 | 3 | 0 | 1 | 3 | 3 | 10 | The blood test shows you have a staph infection in the skin, and it's likely MRSA, which means it doesn't resp |
| f090 | 38 | 3 | 1 | 2 | 5 | 5 | 2 | Take the rifampin on an empty stomach, an hour before breakfast, and expect orange urine; it's harmless. Don't |
| g048 | 61 | 3 | 0 | 0 | 6 | 4 | 4 | You have a concussion, but your scan was normal. Someone should stay with you tonight, and they can wake you o |
| g053 | 68 | 3 | 0 | 0 | 2 | 9 | 3 | For the first two weeks, do your exercises three times a day, ten repetitions each, and walk with the walker e |
| g060 | 73 | 3 | 1 | 0 | 3 | 2 | 4 | Take the levetiracetam 500 milligrams twice a day, morning and evening, at the same times every day. If you mi |
| f002 | 29 | 2 | 0 | 0 | 1 | 1 | 1 | Your A1c came back at 8.2 today, so we're going to add metformin 500 milligrams with dinner and bump it up in  |
| f013 | 30 | 2 | 1 | 0 | 1 | 0 | 0 | We're going to start you on sertraline 25 milligrams at bedtime for a week, then go up to 50, and it can take  |
| f018 | 20 | 2 | 0 | 0 | 2 | 0 | 0 | Your potassium was 5.6, which is a bit high, so skip the salt substitute and we'll recheck it Monday morning. |
| f026 | 22 | 2 | 0 | 0 | 2 | 2 | 1 | Your mammogram showed a spot that's probably benign, but we can't be sure, so we need an ultrasound and maybe  |
| f030 | 24 | 2 | 0 | 1 | 27 | 0 | 12 | Take the insulin glargine 20 units at 10 each night, and if your morning sugar is under 70, call us before the |
| f040 | 21 | 2 | 0 | 1 | 3 | 3 | 1 | It's been three days of diarrhea, so push fluids, small sips often, and call if he hasn't peed in eight hours. |
| f044 | 23 | 2 | 0 | 0 | 4 | 1 | 1 | Your gums are bleeding because of plaque buildup, so floss once a day and come back for a deep cleaning in thr |
| f048 | 80 | 2 | 0 | 1 | 0 | 3 | 1 | You're 28 weeks, so this is when we do the glucose test for gestational diabetes. You'll drink a sweet orange  |
| f049 | 75 | 2 | 16 | 1 | 9 | 8 | 5 | If you start bleeding heavier than a pad an hour, or you pass clots bigger than a golf ball, don't wait, come  |

## Unusable translations, with the judge's reasons

| System | Language | Text | Reason |
|---|---|---|---|
| OpenTalk | Thai | f077 | garbled |
| Haiku | Latin American Spanish | f075 | overwritten with the word placeholder; counted as delivered |
| Haiku | Persian | f049 | too_many_errors |
| Haiku | Traditional Chinese | g054 | wrong_language |
| Qwen | Arabic | g056 | garbled |
| Qwen | Persian | f051 | too_many_errors |
| Qwen | Hindi | f044 | garbled |
| Qwen | Hindi | g017 | garbled |
| Qwen | Hindi | g048 | too_many_errors |
| Qwen | Indonesian | f030 | too_many_errors |
| Qwen | Japanese | f059 | too_many_errors |
| Qwen | Korean | f030 | garbled |
| Qwen | Korean | g010 | garbled |
| Qwen | Russian | g029 | wrong_language |
| Qwen | Thai | g033 | garbled |
| Qwen | Thai | g048 | wrong_language |
| Qwen | Thai | g050 | garbled |
| Qwen | Ukrainian | f051 | too_many_errors |
| Qwen | Vietnamese | f023 | too_many_errors |
| Qwen | Vietnamese | g044 | garbled |
| Qwen | Traditional Chinese | f007 | wrong_language |
| Qwen | Traditional Chinese | f008 | wrong_language |
| Qwen | Traditional Chinese | f011 | wrong_language |
| Qwen | Traditional Chinese | f050 | wrong_language |
| Qwen | Traditional Chinese | f064 | wrong_language |
| Qwen | Traditional Chinese | f066 | wrong_language |
| Qwen | Traditional Chinese | f073 | wrong_language |
| Qwen | Traditional Chinese | g001 | wrong_language |
| Qwen | Traditional Chinese | g004 | wrong_language |
| Qwen | Traditional Chinese | g019 | wrong_language |
| Qwen | Traditional Chinese | g027 | wrong_language |
| Tencent | Korean | f083 | too_many_errors |
| Tencent | Brazilian Portuguese | g025 | too_many_errors |
| Tencent | Thai | f020 | garbled |
| Tencent | Simplified Chinese | f054 | too_many_errors |
| Plain E4B | German | f082 | too_many_errors |
| Plain E4B | Hindi | g014 | too_many_errors |
| Plain E4B | Brazilian Portuguese | g025 | garbled |
| Plain E4B | Russian | f068 | too_many_errors |
| Plain E4B | Ukrainian | f007 | several_versions |
| Plain E4B | Ukrainian | g027 | garbled |
| Plain E4B | Traditional Chinese | f088 | garbled |
| Plain E4B | Traditional Chinese | g006 | garbled |

## OpenTalk's critical errors, every one

| Language | Text | Type | English | Translation | Judge's reading |
|---|---|---|---|---|---|
| Arabic | f068 | logic | I'm cutting the tamsulosin to bedtime only | سأوقف التامسولوسين عند وقت النوم فقط | I will stop tamsulosin at bedtime only, implying take it other times. |
| Arabic | f082 | logic | contractions come closer than five minutes apart | تقترب الانقباضات من بعضها أكثر من 5 دقائق | Contractions more than five minutes apart; reverses the threshold for coming in early |
| Arabic | g025 | numbers | Fast for twelve hours before the test | صم يومًا قبل الفحص لمدة 12 ساعة | fast for a day, then says 12 hours, contradictory |
| Latin American Spanish | g050 | fidelity | but no aspirin | pero no aspire | But do not aspirate/inhale; the aspirin ban is lost. |
| European Spanish | g060 | numbers | until you've gone six months without a seizure | hasta que haya pasado un año sin convulsiones | do not drive until a year without seizures |
| Persian | f017 | logic | don't skip it any other day | در روزهای دیگر هرگز آن را نخورید | On other days never take it |
| Persian | f089 | logic | until it puts out less than 30 milliliters a day for two days | تا زمانی که 2 روز پشت سر هم کمتر از 30 میلی‌لیتر در روز دفع نکند | Drain stays until it does not put out less than 30 mL; reversed. |
| Persian | g017 | fidelity | your baby's temperature | دمای بدنتان | your own body temperature |
| French | f018 | fidelity | skip the salt substitute | ne prenez pas de sel de table sans sel | do not take table salt without salt; the substitute is not named |
| Hindi | g046 | logic | Don't use it if you have a bad cold with heavy congestion until it clears | जब तक भारी बंद नाक के साथ ज़ोर का ज़ुकाम न हो जाए, तब तक इसका इस्तेमाल मत कीजिए | don't use it until you get a bad cold; reversed |
| Japanese | f030 | numbers | insulin glargine 20 units at 10 each night | 毎晩10単位、20単位ずつ注射してください | Inject 10 units and 20 units each night; garbled dose. |
| Japanese | f048 | fidelity | you'll get your Rhogam shot | ルホガム（リゾラグ）の注射 | an injection of a garbled, wrong drug name |
| Japanese | f077 | fidelity | take out half the thyroid, which is called a lobectomy | 甲状腺の半分を切除する「肺葉切除術」 | Removing half the thyroid, called lung lobectomy; wrong organ term. |
| Japanese | g060 | logic | shouldn't drive until you've gone six months without a seizure | 発作が6か月起きなければ、運転しないでください | If no seizure for six months, do not drive; the condition is reversed. |
| Korean | f087 | logic | skip the correction dose of lispro unless you're above 200 | 200을 초과하지 않는 한 리스프로 보정 용량은 투여하지 마십시오 | Do not give lispro unless NOT above 200 (reversed) |
| Korean | g060 | logic | take it as soon as you remember unless it's almost time for the next one, and then skip it | 다음 약 먹을 시간이 거의 되지 않았다면 건너뛰고, 생각난 즉시 드세요 | Skip if next dose is not near, then also take at once. Reversed and contradictory. |
| Polish | f058 | numbers | stop breathing about 32 times an hour | co około 32 minuty przestajesz oddychać | You stop breathing about every 32 minutes, understating the severity. |
| Polish | f085 | fidelity | take nothing by mouth | nic nie przyjmować na czczo | Take nothing on an empty stomach, which could allow eating. |
| Russian | g050 | numbers | Her next visit is at four years | Следующий визит у неё через 4 года | next visit is four years from now, not at age four |
| Thai | f077 | fidelity | which is called a lobectomy | ผ่าตัดตัดกลีบปอด ไม่ใช่การผ่าตัดตัดกลีบปอด | Lung lobe surgery, then 'not lung lobe surgery'; wrong organ and contradictory. |
| Traditional Chinese | f077 | fidelity | which is called a lobectomy | 這叫做肺葉切除術 | Surgery is called lung lobe removal, a different operation. |

## Files

- `translations.csv`: every translation with its scores, source and text.
- `errors.csv`: every error with severity, type, quoted words and the judge's reading.
- `metrics_long.csv`: every number in this report, one per row.
- `report.txt`: the shorter version.