Translation in the Wild

Large Language Models (LLMs) excel in translation among other things, demonstrating competitive performance for many language pairs in zero- and few-shot settings. But unlike dedicated neural machine translation models, LLMs are not trained on any translation-related objective. What explains their remarkable translation abilities? Are these abilities grounded in “incidental bilingualism” (Briakou et al. 2023) in training data? Does instruction tuning contribute to it? Are LLMs capable of aligning and leveraging semantically identical or similar monolingual contents from different corners of the internet that are unlikely to fit in a single context window? I offer some reflections on this topic, informed by recent studies and growing user experience. My working hypothesis is that LLMs’ translation abilities originate in two different types of pre-training data that may be internalized by the models in different ways. I discuss the prospects for testing the “duality” hypothesis empirically and its implications for reconceptualizing translation, human and machine, in the age of deep learning.

💡 Research Summary

The paper “Translation in the Wild” investigates why large language models (LLMs) display surprisingly strong translation capabilities despite never being trained on a dedicated translation objective. The author proposes a “duality” hypothesis: LLMs acquire translation ability from two distinct types of pre‑training data and learning processes, termed “Local learning” and “Global learning.”

Local learning refers to bilingual signals that appear within a single training context window – for example, an English sentence followed shortly by its French translation in the raw corpus. These short‑range, explicitly aligned pairs allow the model to learn a direct source‑target mapping much like a conventional neural machine translation (NMT) system.

Global learning, by contrast, exploits the massive amount of monolingual content scattered across the internet. Many documents in different languages convey essentially the same information (e.g., Wikipedia articles, news reports, scientific abstracts) but are not aligned at the sentence level and rarely co‑occur within a single context window. The hypothesis is that, through the sheer scale of parameters (trillions) and the breadth of pre‑training data (hundreds of billions of tokens), LLMs develop mechanisms to locate, retrieve, and align these semantically similar monolingual fragments across distant parts of the training set. This meta‑learning process yields a cross‑lingual representation space that can be leveraged for translation even when no explicit parallel evidence is present.

The paper reviews the historical evolution of translation technologies, from rule‑based systems to statistical MT, to the transformer‑based NMT era, and finally to the emergence of LLMs. A detailed comparison (Section 3) highlights differences in architecture (encoder‑decoder vs. decoder‑only), model size (10⁸–10¹⁰ vs. 10¹²–10¹³ parameters), training data (narrow parallel corpora vs. massive heterogeneous multilingual web text), and objectives (explicit translation loss vs. next‑token prediction).

Recent applications of LLMs to translation tasks are surveyed (Section 4). Zero‑ and few‑shot prompting, low‑resource language support, pivot‑language strategies, chain‑of‑thought prompting, and the use of multilingual latent representations for cross‑lingual tasks all demonstrate that LLMs can perform sophisticated translation‑related operations without fine‑tuning. Section 5 discusses theoretical work on how LLMs translate: in‑context translation, cross‑lingual representation alignment, emergent pivoting through English, and the debate over language‑agnostic versus language‑specific representations.

Section 6 is the core of the argument. The author downplays the role of instruction tuning, citing recent studies that show minimal performance gains for translation after instruction fine‑tuning. Instead, “incidental bilingualism” – the presence of occasional parallel sentences in web corpora – provides the raw material for Local learning. Global learning is argued to be the dominant factor for the observed zero‑shot performance, especially for language pairs that rarely appear together in the training data. The paper outlines empirical implications: (i) translation style (literal vs. idiomatic) may reveal the balance between Local and Global influences; (ii) scaling trends should show increasing Global effects; (iii) generalization to unseen language pairs and domains would be a hallmark of Global learning.

Section 7 proposes concrete experimental designs to test the duality hypothesis: selective ablation of known parallel data, synthetic probing tasks that isolate Local vs. Global signals, human evaluation focusing on error typology, and the use of metrics such as BLEU, COMET, and human preference scores.

Section 8 broadens the discussion to the philosophy of translation. The author argues that translation should be reconceptualized as a pluralistic process, mirroring how human translators combine local resources (glossaries, translation memories) with global knowledge (cultural context, world knowledge). The opacity of LLMs makes interpretability crucial; mechanistic studies could bridge translation studies and distributional semantics.

The conclusion reiterates the duality hypothesis, emphasizes its testability, and calls for future work that integrates data‑centric analysis, meta‑learning techniques, and human‑AI collaborative workflows. Overall, the paper offers a compelling framework for understanding why LLMs translate so well and sets an agenda for systematic empirical validation.