XCR-Bench: A Multi-Task Benchmark for Evaluating Cultural Reasoning in LLMs

Cross-cultural competence in large language models (LLMs) requires the ability to identify Culture-Specific Items (CSIs) and to adapt them appropriately across cultural contexts. Progress in evaluating this capability has been constrained by the scarcity of high-quality CSI-annotated corpora with parallel cross-cultural sentence pairs. To address this limitation, we introduce XCR-Bench, a Cross(X)-Cultural Reasoning Benchmark consisting of 4.9k parallel sentences and 1,098 unique CSIs, spanning three distinct reasoning tasks with corresponding evaluation metrics. Our corpus integrates Newmark’s CSI framework with Hall’s Triad of Culture, enabling systematic analysis of cultural reasoning beyond surface-level artifacts and into semi-visible and invisible cultural elements such as social norms, beliefs, and values. Our findings show that state-of-the-art LLMs exhibit consistent weaknesses in identifying and adapting CSIs related to social etiquette and cultural reference. Additionally, we find evidence that LLMs encode regional and ethno-religious biases even within a single linguistic setting during cultural adaptation. We release our corpus and code to facilitate future research on cross-cultural NLP.

💡 Research Summary

The paper introduces XCR‑Bench, a comprehensive benchmark designed to evaluate cross‑cultural reasoning capabilities of large language models (LLMs). Recognizing the scarcity of high‑quality, CSI‑annotated parallel corpora, the authors construct a dataset comprising 4,900 parallel sentences and 1,098 unique Culture‑Specific Items (CSIs). They integrate Newmark’s CSI taxonomy with Hall’s Triad of Culture (Visible, Semi‑visible, Invisible), thereby enabling analysis that goes beyond surface linguistic cues to include deeper social norms, beliefs, and values.

Data construction proceeds in two phases. First, CSIs are extracted from structured cultural knowledge bases—CANDLE and the Cultural Atlas—focusing on US/UK culture. For each CSI, three candidate sentences are generated using GPT‑4o, Claude‑3.7‑Sonnet, and DeepSeek‑R1; human annotators select the most accurate, fluent, and realistic version and mark the CSI with tags. Annotators then assign each item to one of Newmark’s four categories and map it to Hall’s cultural level, achieving substantial inter‑annotator agreement (Cohen’s κ ≈ 0.66).

Second, culturally adapted counterparts are created for four target cultures: Chinese, Arabic, and Bengali (split into West Bengal and Bangladesh variants). Native annotators produce adaptations following a detailed protocol that distinguishes four equivalence types—direct, functional, neutral, and non‑transferable—allowing multiple valid adaptations per item. Disagreements are resolved through discussion and expert adjudication, reflecting the inherent subjectivity of cultural equivalence.

XCR‑Bench defines three evaluation tasks: (1) CSI Identification, where models must recover the CSI term from a sentence with tags removed; (2) CSI Prediction, requiring generation of an appropriate CSI given a cultural context; and (3) CSI Adaptation, demanding transformation of a source‑culture sentence into a target‑culture version while preserving meaning and cultural appropriateness. Each task is paired with bespoke metrics (accuracy, F1, BLEU, cultural suitability scores) to capture both lexical correctness and deeper cultural reasoning.

The authors evaluate several state‑of‑the‑art LLMs (GPT‑4o, Claude‑3.7, LLaMA‑2, Gemini). Results reveal consistent weaknesses in CSI identification and adaptation, especially for items related to social etiquette (e.g., wedding rings, dating apps) and cultural references (e.g., traditional foods, idioms) that reside in Hall’s semi‑visible and invisible layers. Moreover, intra‑lingual bias emerges: models perform unevenly across the two Bengali variants, indicating regional and ethno‑religious biases even when the language is identical.

Overall, XCR‑Bench provides the first publicly released corpus that aligns CSIs with Hall’s cultural model, supports both intra‑ and inter‑lingual reasoning, and offers a structured framework for probing cultural competence beyond translation. By releasing the data, annotation guidelines, and evaluation scripts, the authors enable the community to develop more culturally aware LLMs and to systematically address embedded biases.