How Should We Model the Probability of a Language?

Of the over 7,000 languages spoken in the world, commercial language identification (LID) systems only reliably identify a few hundred in written form. Research-grade systems extend this coverage under certain circumstances, but for most languages coverage remains patchy or nonexistent. This position paper argues that this situation is largely self-imposed. In particular, it arises from a persistent framing of LID as decontextualized text classification, which obscures the central role of prior probability estimation and is reinforced by institutional incentives that favor global, fixed-prior models. We argue that improving coverage for tail languages requires rethinking LID as a routing problem and developing principled ways to incorporate environmental cues that make languages locally plausible.

💡 Research Summary

The paper opens by highlighting a stark disparity: while more than 7,000 languages exist worldwide, commercial language identification (LID) systems reliably support only a few hundred, and even research‑grade systems leave most languages uncovered. The authors trace this gap to two intertwined structural issues. First, LID is habitually framed as a de‑contextualized text‑classification task that maps raw text directly to a label drawn from a fixed, global hypothesis space. This framing encourages models that excel on benchmark datasets but falter in real‑world deployments where the set of plausible languages varies dramatically across domains. Second, the prevailing Bayesian formulation of LID—(P(\ell|X) \propto P(X|\ell)P(\ell))—relies on a prior (P(\ell)) estimated from global frequency counts. Because language frequencies follow a heavy‑tailed distribution, rare languages receive priors near zero, making them virtually invisible regardless of how well their likelihood models fit the data.

The authors illustrate the problem with a toy example: even if a rare language’s likelihood is two orders of magnitude higher than English’s for a given snippet, the overwhelming global prior for English still dominates the posterior, leading to systematic misclassification of rare languages. Common mitigation—oversampling rare classes or undersampling dominant ones—only reshapes the training distribution; at web scale, a modest false‑positive rate (e.g., 0.01 %) translates into millions of erroneous language tags, rendering large‑scale corpora unusable. Sources of false positives include emojis, mis‑rendered PDFs, non‑Unicode fonts, accidental n‑gram overlaps, and boilerplate “ANT‑SPEAK” text.

Recognizing that global priors are often irrelevant in specific contexts, the paper advocates for “local priors” derived from environmental cues such as script, geographic metadata, social‑network signals, or user feedback. For instance, in a dedicated online community for Louisiana Creole, the effective prior for that language can be orders of magnitude higher than its global frequency, yet current LID models treat its prior as zero because the label set is fixed at training time. The authors note that most public LID datasets (Wikipedia, the Bible, newswire) lack such contextual metadata, making it difficult to train or evaluate context‑aware priors. Moreover, modern discriminative models (fastText, CLD3) are deeply embedded in processing pipelines and lack interfaces for external hints, unlike older per‑class models (e.g., CLD2) that allowed users to supply script or domain information.

To overcome these limitations, the authors propose reconceptualizing LID as a routing problem. The first routing stage would use cheap, high‑level cues (script detection, URL domain, geolocation) to prune the candidate language set dynamically. The second stage would apply a fine‑grained text model only within this reduced set, thereby preserving discriminative power while avoiding the dominance of global priors. This two‑stage approach effectively decouples language detection from full downstream support, allowing rare or emerging languages to be identified even when translation or other services are unavailable.

Two case studies illustrate the stakes. The first examines Louisiana Creole, a critically endangered language with a modest online presence. Commercial systems misclassify it as French or Bambara, and manual override options are only offered for fully supported languages, leaving users unable to correct the initial LID error. The second looks at the extinct Mediterranean Lingua Franca; despite its historical relevance, it is absent from any global label inventory, so all state‑of‑the‑art models ignore it entirely.

Finally, the paper critiques institutional incentives that reward improvements on fixed‑label benchmarks, reinforcing the de‑contextualized paradigm. The authors argue that to expand coverage for tail languages, the community must (1) collect and share data with rich provenance metadata, (2) adopt evaluation metrics that penalize false positives at scale and reward context‑aware accuracy, and (3) develop modular LID pipelines that can ingest external priors at inference time. In sum, the work calls for a shift from a static, globally‑biased classification mindset to a dynamic, context‑driven routing framework that respects the true probabilistic nature of language occurrence.