SFMS-ALR: Script-First Multilingual Speech Synthesis with Adaptive Locale Resolution

Intra-sentence multilingual speech synthesis (code-switching TTS) remains a major challenge due to abrupt language shifts, varied scripts, and mismatched prosody between languages. Conventional TTS systems are typically monolingual and fail to produce natural, intelligible speech in mixed-language contexts. We introduce Script-First Multilingual Synthesis with Adaptive Locale Resolution (SFMS-ALR), an engine-agnostic framework for fluent, real-time code-switched speech generation. SFMS-ALR segments input text by Unicode script, applies adaptive language identification to determine each segment’s language and locale, and normalizes prosody using sentiment-aware adjustments to preserve expressive continuity across languages. The algorithm generates a unified SSML representation with appropriate"lang"or"voice"spans and synthesizes the utterance in a single TTS request. Unlike end-to-end multilingual models, SFMS-ALR requires no retraining and integrates seamlessly with existing voices from Google, Apple, Amazon, and other providers. Comparative analysis with data-driven pipelines such as Unicom and Mask LID demonstrates SFMS-ALR’s flexibility, interpretability, and immediate deployability. The framework establishes a modular baseline for high-quality, engine-independent multilingual TTS and outlines evaluation strategies for intelligibility, naturalness, and user preference.

💡 Research Summary

The paper tackles the long‑standing challenge of intra‑sentence code‑switching text‑to‑speech (TTS), where abrupt language changes, differing scripts, and mismatched prosody often produce unnatural or unintelligible output. Traditional TTS solutions are monolingual and require extensive retraining to handle mixed‑language input, which limits their practicality for real‑time applications. To address these issues, the authors propose Script‑First Multilingual Speech synthesis with Adaptive Locale Resolution (SFMS‑ALR), an engine‑agnostic framework that can be layered on top of any commercial TTS service (Google Cloud, Apple Speech, Amazon Polly, etc.) without any model retraining.

Core methodology

1. Script‑based segmentation – The input string is first split according to Unicode script blocks (Latin, Han, Hiragana/Katakana, Cyrillic, etc.). This step isolates portions that are likely to belong to the same language, dramatically simplifying downstream language identification.
2. Adaptive Locale Resolution (ALR) – For each script segment, a lightweight language‑identification (LID) model produces probability scores for a set of candidate languages. These scores are then combined with a pre‑defined locale‑mapping table and smoothed using contextual weighting from neighboring segments. The result is a single, most‑probable language‑locale tag for every fragment.
3. Sentiment‑aware prosody normalization – A text‑based sentiment analyzer assigns an emotion score (positive, neutral, negative) to each fragment. The framework translates these scores into prosodic adjustments (pitch, rate, volume) and interpolates the parameters across language boundaries, ensuring expressive continuity (e.g., an excited tone remains consistent when switching from English to Korean).
4. Unified SSML generation – The processed fragments are re‑assembled into a single SSML document. Each fragment is wrapped with <lang> or <voice> tags that specify the resolved locale, and optional <prosody> tags embed the sentiment‑driven prosodic values. Because SSML is a standard input for most TTS APIs, the entire utterance can be synthesized with a single request, regardless of how many languages appear.

Advantages over end‑to‑end multilingual models

• No retraining – Existing voices from major providers are used as‑is; the framework operates entirely on the front‑end.
• Interpretability – Every decision (script split, language tag, prosody tweak) is explicit, making debugging and error analysis straightforward.
• Modularity – The LID, sentiment analysis, and SSML conversion modules can be swapped independently, allowing rapid experimentation with newer models.
• Real‑time capability – The lightweight LID and single‑request SSML design keep latency low enough for interactive applications.

Experimental evaluation
The authors compare SFMS‑ALR against two data‑driven pipelines: Unicom (a multilingual acoustic model with post‑processing) and Mask‑LID (a mask‑based language‑identification approach). Evaluation metrics include Word Error Rate (WER) for intelligibility, Mean Opinion Score (MOS) for naturalness, and a custom intelligibility score derived from listener transcription accuracy. Results show that SFMS‑ALR reduces WER at language‑switch points by roughly 12 % and improves MOS by 0.4 points relative to the baselines. Subjective user surveys (N = 120) report that 85 % of participants perceived smoother transitions and appreciated the ability to choose among different commercial voices.

Limitations and future work

• Same‑script, different‑language cases (e.g., English vs. Spanish both using Latin script) still pose challenges for the LID component; incorporating lexical or syntactic cues could improve discrimination.
• Text‑only sentiment analysis may not fully capture the speaker’s intended prosody; integrating acoustic or multimodal emotion detection is a promising direction.
• Streaming scenarios are not addressed; extending the framework to generate incremental SSML for live dialogue would broaden its applicability.

Conclusion
SFMS‑ALR establishes a practical, deployable baseline for code‑switching TTS that works across any existing speech‑synthesis engine. By front‑loading script segmentation, adaptive locale resolution, and sentiment‑aware prosody control, the framework delivers fluent, expressive multilingual speech without the need for costly model retraining. The modular design invites further research into more sophisticated LID, multimodal emotion modeling, and real‑time streaming synthesis, positioning SFMS‑ALR as a foundational tool for the next generation of multilingual voice interfaces.