BhashaSetu: Cross-Lingual Knowledge Transfer from High-Resource to Extreme Low-Resource Languages

Despite remarkable advances in natural language processing, developing effective systems for low-resource languages remains a formidable challenge, with performances typically lagging far behind high-resource counterparts due to data scarcity and insufficient linguistic resources. Cross-lingual knowledge transfer has emerged as a promising approach to address this challenge by leveraging resources from high-resource languages. In this paper, we investigate methods for transferring linguistic knowledge from high-resource languages to low-resource languages, where the number of labeled training instances is in hundreds. We focus on sentence-level and word-level tasks. We introduce a novel method, GETR (Graph-Enhanced Token Representation) for cross-lingual knowledge transfer along with two adopted baselines (a) augmentation in hidden layers and (b) token embedding transfer through token translation. Experimental results demonstrate that our GNN-based approach significantly outperforms existing multilingual and cross-lingual baseline methods, achieving 13 percentage point improvements on truly low-resource languages (Mizo, Khasi) for POS tagging, and 20 and 27 percentage point improvements in macro-F1 on simulated low-resource languages (Marathi, Bangla, Malayalam) across sentiment classification and NER tasks respectively. We also present a detailed analysis of the transfer mechanisms and identify key factors that contribute to successful knowledge transfer in this linguistic context.

💡 Research Summary

The paper addresses the pressing challenge of building effective NLP systems for extreme low‑resource languages, where only a few hundred labeled examples are available. While multilingual pre‑trained models such as XLM‑R, mmBERT, and recent parameter‑efficient adapters (LoRA, AdaMergeX) have shown impressive cross‑lingual transfer in moderate low‑resource settings, they still struggle when the target language provides as few as 100 training instances. To bridge this gap, the authors propose a comprehensive framework named BhashaSetu, comprising two adapted baselines and one novel method.

The first baseline, Hidden Augmentation Layers (HAL), creates synthetic training pairs by linearly mixing the CLS representations of high‑resource (HR) and low‑resource (LR) language examples using a mixing coefficient α. Labels are mixed in the same proportion, producing soft targets that are optimized with KL‑divergence loss. Empirically, α values between 0.1 and 0.4 preserve LR language characteristics while still injecting HR knowledge.

The second baseline, Token Embedding Transfer through Translation (TET), leverages bilingual dictionaries or manually curated translations to map each LR token to its HR counterpart. The HR token embeddings (from a large pre‑trained model) are averaged and assigned as the initial embedding for the LR token. This initialization dramatically reduces the risk of over‑fitting in the LR fine‑tuning stage, especially when the LR vocabulary is tiny. The authors provide a clear algorithmic pipeline and discuss practical considerations for languages lacking ready‑made dictionaries.

The core contribution is GETR (Graph‑Enhanced Token Representation). For each training batch, a token‑level undirected graph is constructed where nodes correspond to tokens and edges encode sequential adjacency within sentences. A Graph Convolutional Network (GCN) or Graph Attention Network (GAT) processes the reshaped token embeddings, producing refined representations that are fed back into the Transformer as the Query and Key matrices while leaving the Value computation unchanged. This design enables dynamic, fine‑grained sharing of contextual information across languages at the token level, allowing shared tokens (e.g., “was” in English and a low‑resource language) to benefit from both linguistic contexts. Multiple GNN layers can be stacked; the authors find 2‑3 layers strike a balance between transfer strength and computational overhead.

Experiments cover three NLP tasks: part‑of‑speech (POS) tagging, sentiment classification, and named‑entity recognition (NER). Real ultra‑low‑resource languages Mizo and Khasi (≈500 annotated sentences) serve as true low‑resource testbeds, while Marathi, Bangla, and Malayalam are treated as simulated low‑resource languages by restricting their training data to 100 instances. High‑resource counterparts are English and Hindi. Baselines include multilingual models (XLM‑R, mmBERT), adapter‑based fine‑tuning, and existing data‑augmentation techniques.

Results show that GETR consistently outperforms both baselines and all multilingual baselines. Specifically, GETR yields a 13‑point macro‑F1 gain on POS tagging for Mizo and Khasi, and 20‑point (sentiment) and 27‑point (NER) macro‑F1 improvements on the simulated low‑resource languages. Ablation studies reveal that (i) the mixing coefficient α critically balances LR preservation versus HR infusion, (ii) 2‑3 GNN layers provide the best trade‑off, and (iii) the quality of the translation dictionary directly impacts TET performance.

The authors also discuss limitations: (a) TET requires at least a minimal bilingual lexicon, which may be unavailable for many truly low‑resource languages, and (b) constructing token graphs for large batches can be memory‑intensive. Future work is suggested on unsupervised token alignment and efficient graph sampling to improve scalability.

In summary, BhashaSetu introduces a versatile, graph‑centric approach to cross‑lingual knowledge transfer that markedly improves performance on tasks with only a few hundred labeled examples, demonstrating that dynamic token‑level interaction via GNNs is a powerful complement to existing hidden‑space augmentation and embedding‑transfer techniques.