Utilizing TTS Synthesized Data for Efficient Development of Keyword Spotting Model

This paper explores the use of TTS synthesized training data for KWS (keyword spotting) task while minimizing development cost and time. Keyword spotting models require a huge amount of training data to be accurate, and obtaining such training data can be costly. In the current state of the art, TTS models can generate large amounts of natural-sounding data, which can help reducing cost and time for KWS model development. Still, TTS generated data can be lacking diversity compared to real data. To pursue maximizing KWS model accuracy under the constraint of limited resources and current TTS capability, we explored various strategies to mix TTS data and real human speech data, with a focus on minimizing real data use and maximizing diversity of TTS output. Our experimental results indicate that relatively small amounts of real audio data with speaker diversity (100 speakers, 2k utterances) and large amounts of TTS synthesized data can achieve reasonably high accuracy (within 3x error rate of baseline), compared to the baseline (trained with 3.8M real positive utterances).

💡 Research Summary

The paper investigates how to dramatically reduce the data collection cost for keyword spotting (KWS) systems by leveraging large‑scale text‑to‑speech (TTS) synthesis. Traditional production‑grade KWS models rely on millions of real utterances to cover the wide range of speakers, accents, and acoustic conditions. Collecting and annotating such data is expensive and time‑consuming, especially under privacy constraints. Recent advances in neural TTS, capable of generating natural‑sounding speech from arbitrary text, open the possibility of substituting a substantial portion of the real training set with synthetic audio. However, synthetic speech typically lacks the full diversity of real recordings and may contain artifacts that hurt model generalisation.

The authors propose a systematic framework that combines (1) a text generator that creates both positive (keyword‑containing) and negative (keyword‑free) utterances, (2) two state‑of‑the‑art TTS engines—V irtuoso (multilingual, 726 pretrained voices, prosody control via punctuation symbols) and an AudioLM‑based TTS that can clone a target speaker’s voice from a short audio sample—and (3) a set of mixing strategies that vary the proportion of real and synthetic data, the number of distinct speakers, and the number of utterances per speaker.

Text generator – The generator receives a keyword definition (prefix + key name, e.g., “Hey Google”) and a random query corpus. It assembles templates such as “{prefix} {key_name} {query}” for positive examples and filters the keyword out of random sentences for negatives. To increase acoustic variability, it randomly inserts prosody control symbols (parentheses, question marks, exclamation points) that V irtuoso interprets as cues for slower speech, higher pitch, louder volume, etc. This simple but effective augmentation expands the distribution of the synthetic set without requiring additional audio resources.

TTS engines – V irtuoso supplies a large pool of high‑quality synthetic voices across many languages, enabling the creation of accented English or non‑English variants of the same keyword phrase. The AudioLM‑based model can generate speech that preserves the timbre and prosodic style of any reference speaker, allowing the authors to simulate a “personalized” voice for each synthetic utterance. By mixing both engines, the final synthetic corpus contains 7.5 M positive and 5.1 M negative utterances, covering a broad spectrum of speaker identities, languages, and prosodic patterns.

KWS model architecture – The baseline model follows the streaming‑friendly design used in prior Google work: 40‑dimensional filter‑bank features are stacked (three consecutive frames) to form 120‑dimensional vectors every 20 ms. The network consists of seven factored convolution (SVDF) layers followed by three bottleneck projection layers, split into an encoder that outputs phoneme‑like embeddings and a decoder that predicts a binary presence/absence label. Training uses a weighted sum of frame‑wise cross‑entropy (L_CE) and a max‑pool loss (L_MP) that encourages correct detection of the keyword’s endpoint.

Experimental setup – Real data consists of 3.8 M positive “Hey/Ok Google” utterances and 14.1 M negative utterances collected under Google’s privacy policies. Synthetic data is generated as described above. Evaluation is performed on a held‑out real test set, measuring false‑reject rate (FRR) while fixing the false‑accept rate at 0.133 per hour (a typical operational target).

Key experiments

1. Baseline mixing – Training on synthetic data alone yields very high FRR (≈46 %). Adding the large real negative set (≈11 M utterances) reduces FRR to ≈17 % for all synthetic variants, showing that even a modest amount of real background speech dramatically stabilises the model.

2. Incremental real positives – With all synthetic data present, the authors gradually add real positive utterances (0 → 100 k). FRR drops monotonically: 46 % → 17 % (with negatives) → 2.46 % when the full 3.8 M real positives are used. Notably, 100 k real positives (≈0.1 % of the full set) combined with synthetic data achieve an FRR of 9.94 %, roughly three times the baseline error.

3. Speaker‑count variation – Keeping synthetic data fixed, the authors sample 10 utterances per speaker and increase the number of distinct speakers from 10 to 300. FRR improves sharply up to about 100 speakers and then plateaus, indicating that speaker diversity is far more valuable than additional utterances from the same speakers.

4. Utterances‑per‑speaker variation – With 100 speakers fixed, the number of utterances per speaker is increased (10 → 100). The FRR gains are modest, confirming the earlier finding that after a certain speaker count, extra repetitions provide diminishing returns.

Results summary – The best mixed model (all synthetic + 100 k real positives + full real negatives) attains an FRR of 2.46 %, slightly better than the pure‑real baseline (3.17 %). More importantly, a model trained with only 1 k real positive utterances (100 speakers × 10 utterances) and the full synthetic corpus reaches an FRR of ≈9.94 %, i.e., within a factor of three of the baseline while using less than 0.03 % of the original real positive data. This demonstrates that a tiny, highly diverse real subset can substitute for massive data collection when paired with high‑quality TTS.

Discussion and limitations – While the approach cuts data acquisition costs dramatically, synthetic speech still cannot fully emulate real‑world acoustic variability such as microphone characteristics, background noises, and spontaneous prosodic quirks. The prosody control used (simple punctuation symbols) is coarse; more sophisticated conditioning (e.g., explicit pitch contours or style tokens) could further close the gap. Moreover, the study relies on a large pool of real negative data; future work should explore whether synthetic negatives can also be effective, perhaps via adversarial generation. Domain‑adaptation techniques (e.g., GAN‑based refinement of synthetic audio, feature‑level alignment) are promising avenues to reduce the residual distribution mismatch.

Conclusion – The paper provides strong empirical evidence that a strategic combination of a small, speaker‑diverse real corpus and massive TTS‑generated audio can yield KWS models with near‑baseline performance at a fraction of the data‑collection cost. The methodology—text‑template generation, prosody‑aware TTS, and systematic mixing experiments—offers a practical recipe for industry practitioners aiming to launch new wake‑word products quickly and economically. Future research should focus on refining synthetic audio quality, reducing reliance on real negatives, and applying the same paradigm to multilingual or multi‑keyword scenarios.