"Be My Cheese?": Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs

We present a large-scale human evaluation benchmark for assessing cultural localisation in machine translation produced by state-of-the-art multilingual large language models (LLMs). Existing MT benchmarks emphasise token-level and grammatical accuracy, but of ten overlook pragmatic and culturally grounded competencies required for real-world localisation. Building on a pilot study of 87 translations across 20 languages, we evaluate 7 multilingual LLMs across 15 target languages with 5 native-speaker raters per language. Raters scored both full-text translations and segment-level instances of culturally nuanced language (idioms, puns, holidays, and culturally embedded concepts) on an ordinal 0-3 quality scale; segment ratings additionally included an NA option for untranslated segments. Across full-text evaluations, mean overall quality is modest (1.68/3): GPT-5 (2.10/3), Claude Sonnet 3.7 (1.97/3), and Mistral Medium 3.1 (1.84/3) form the strongest tier with fewer catastrophic failures. Segment-level results show sharp category effects: holidays (2.20/3) and cultural concepts (2.19/3) translate substantially better than idioms (1.65/3) and puns (1.45/3), and idioms are most likely to be left untranslated. These findings demonstrate a persistent gap between grammatical adequacy and cultural resonance. To our knowledge, this is the first multilingual, human-annotated benchmark focused explicitly on cultural nuance in translation and localisation, highlighting the need for culturally informed training data, improved cross-lingual pragmatics, and evaluation paradigms that better reflect real-world communicative competence.

💡 Research Summary

The paper introduces a large‑scale human‑evaluation benchmark specifically designed to measure cultural localisation in machine translation (MT) produced by state‑of‑the‑art multilingual large language models (LLMs). While existing MT benchmarks focus on token‑level lexical accuracy, grammatical correctness, and automatic metrics such as BLEU, COMET, or BLEURT, they largely ignore pragmatic and culturally grounded competencies that are essential for real‑world localisation tasks (e.g., marketing, customer engagement, brand messaging).

Building on a pilot study of 87 translations across 20 languages, the authors scale up to a dataset that includes seven publicly available multilingual LLMs, fifteen target languages, and five native‑speaker raters per language (total N = 75 raters). The source material consists of five English e‑commerce marketing emails that deliberately contain culturally nuanced expressions: idioms, puns, holiday references, and culturally specific concepts (e.g., “koozies”, “sweetheart”, “zero‑waste”). For each language, four instances of each category are extracted, yielding 13,125 segment‑level annotations.

Evaluation proceeds in two layers. First, raters score the full translated email on a four‑point scale for content fidelity, style fidelity, audience appropriateness, and overall quality. Second, they assess each pre‑selected segment on a 0‑3 ordinal scale, with an additional “NA” option indicating the segment was left untranslated. This design enables a direct comparison between holistic translation quality and fine‑grained failure modes.

Statistical analysis uses cumulative link mixed models (CLMM) with random intercepts for annotator and segment, appropriate for ordinal outcomes. Fixed effects include model, language, and segment category, as well as their interactions. The final model converges (logLik = ‑14,411.63; AIC = 28,965.26) and shows that segment‑level variance (SD = 1.76) exceeds annotator variance (SD = 0.70), indicating that the intrinsic difficulty of a cultural segment drives most rating variability. Inter‑rater reliability is measured with Krippendorff’s α and Gwet’s AC2; full‑text ratings achieve moderate agreement, while idioms and puns exhibit lower agreement, reflecting higher subjectivity in evaluating figurative language.

Key findings:

1. Overall full‑text quality is modest (mean = 1.68/3). GPT‑5 leads with 2.10/3, followed closely by Claude Sonnet 3.7 (1.97) and Mistral Medium 3.1 (1.84). Aya Expanse 8B is a clear outlier at 1.09. CLMM confirms a significant main effect of model; GPT‑5, Claude Sonnet 3.7, and Mistral Medium 3.1 form a statistically indistinguishable top tier.

2. Segment‑level performance varies dramatically by category. Holidays (2.20) and cultural concepts (2.19) receive the highest scores, while idioms (1.65) and puns (1.45) lag far behind. Idioms are most frequently left untranslated (NA), followed by puns; holidays and cultural concepts are rarely omitted.

3. Language‑level analysis shows higher scores for Romance and East‑Asian languages, and lower scores for low‑resource languages such as Afrikaans, Swahili, and Urdu, suggesting data‑resource disparities affect cultural competence.

4. Model‑category interactions reveal that GPT‑5 and Claude Sonnet 3.7 consistently outperform other models across all categories, whereas Aya Expanse 8B suffers both lower quality scores and higher omission rates.

The authors argue that the gap between grammatical adequacy (often captured by existing metrics) and cultural resonance is substantial. Current LLMs, despite massive multilingual pre‑training, still default to literal or “copy‑English” strategies when faced with non‑literal, figurative language. The benchmark therefore highlights a critical weakness: the inability to reliably translate idiomatic and humorous content, which is essential for authentic localisation.

Implications:

• Training data must be enriched with culturally diverse, pragmatically annotated examples to teach models how to adapt idioms, jokes, and culturally bound references.
• Evaluation frameworks for MT should incorporate cultural‑appropriateness dimensions, moving beyond surface‑level metrics.
• Model developers may consider fine‑tuning on domain‑specific localisation corpora or integrating external cultural knowledge bases.

In sum, this work delivers the first multilingual, human‑annotated benchmark that explicitly measures cultural nuance in translation, provides a thorough empirical analysis of failure modes across models, languages, and content types, and calls for a paradigm shift toward culturally informed MT research and development.