GraphemeAug: A Systematic Approach to Synthesized Hard Negative Keyword Spotting Examples

Spoken Keyword Spotting (KWS) is the task of distinguishing between the presence and absence of a keyword in audio. The accuracy of a KWS model hinges on its ability to correctly classify examples close to the keyword and non-keyword boundary. These boundary examples are often scarce in training data, limiting model performance. In this paper, we propose a method to systematically generate adversarial examples close to the decision boundary by making insertion/deletion/substitution edits on the keyword’s graphemes. We evaluate this technique on held-out data for a popular keyword and show that the technique improves AUC on a dataset of synthetic hard negatives by 61% while maintaining quality on positives and ambient negative audio data.

💡 Research Summary

The paper tackles a persistent weakness in modern keyword‑spotting (KWS) systems: the inability to reliably reject audio that is acoustically close to the target keyword but does not contain it. Existing approaches either mine real‑world confusable utterances from large corpora or hand‑craft a limited list of phonetic look‑alikes. Both strategies suffer from data scarcity, especially for rare proper nouns, unusual pronunciations, or emerging slang that may later become false triggers.

To address this, the authors introduce GraphemeAug, a systematic algorithm that generates synthetic “hard negative” examples by editing the graphemes (written characters) of the target keyword. The algorithm applies three elementary edit operations—single‑character insertion, deletion, and substitution (restricted to the same phonetic class, i.e., vowel ↔ vowel or consonant ↔ consonant). By recursively exploring all combinations up to a chosen Levenshtein distance (the paper experiments with distances 1, 2, and 3), GraphemeAug can produce thousands to tens of thousands of unique confusable strings. Because the method ignores linguistic constraints, many outputs are non‑standard spellings, yet the underlying assumption is that a small orthographic change typically yields only a subtle phonetic shift, creating audio that is perceptually similar to the true keyword.

The generated transcripts are turned into audio using an AudioLM‑based text‑to‑speech (TTS) system. Two TTS modes are evaluated: (1) a vanilla mode that samples a random synthetic speaker, and (2) a style‑transfer mode that conditions the synthesis on a real‑world reference utterance, thereby copying its prosody and speaker characteristics. Experiments show that style‑transfer TTS consistently outperforms the vanilla version, improving the area under the ROC curve (AUC) by roughly 22 % across a range of false‑accept rates. This demonstrates that high‑fidelity, speaker‑matched synthetic audio is crucial for effective KWS training.

Data preparation proceeds in three stages. First, a baseline dataset is built from 13 real‑speech corpora (≈600 k utterances each). Each utterance contains a placeholder keyword; the placeholder is replaced with the target phrase (“Hey Google”) to create positive examples, or removed entirely to create non‑keyword negatives. The resulting audio is re‑synthesized with the TTS pipeline, augmented with 25 room‑simulation and noise conditions, and converted to 120‑dimensional filter‑bank features. This yields 195 M positive and 195 M negative examples, balanced 1:1.

Second, the GraphemeAug confusables are injected: a set of 10 k unique edited strings (edit distance 3) is generated, and 10 % of the negative training examples are replaced with these synthetic confusables. The authors also vary the number of unique confusables (10, 100, 1 k, 10 k) and the edit distance to study scaling effects.

Third, evaluation data comprise four collections: (a) real‑speech positives (61 736 examples) and negatives (20 190 examples), (b) a hand‑picked set of 3 779 real confusable utterances (“eval‑real‑conf”), (c) a synthetic confusable set of 9 595 examples generated with edit distance 3 (“eval‑ed3”), and (d) the synthetic baseline negatives used for training.

The model architecture is a streaming‑friendly two‑stage encoder‑decoder built around seven factored convolution (SVDF) layers and three bottleneck projections, totaling ~320 k parameters—similar to prior KWS work.

Key experimental findings:

1. Style‑transfer TTS superiority – Using style‑transfer audio for both positives and negatives raises AUC from 97.4 % (standard TTS) to 99.65 % (style‑transfer). Consequently, all subsequent experiments employ the style‑transfer pipeline.

2. Hard‑negative confusables dramatically improve robustness – Adding the 10 k edit‑distance‑3 confusables (10 % of negatives) lifts AUC on the synthetic hard‑negative test set by 61 % (from 96.9 % to 98.8 %). Crucially, performance on the real‑speech positive set remains essentially unchanged (99.63 % vs 99.65 %).

3. Scale matters more than edit depth – Increasing the number of unique confusables from 10 to 10 k yields a monotonic AUC gain (≈58 % improvement). Varying edit distance (1 vs 3) has a modest effect; larger distances produce slightly higher AUC but risk over‑penalizing near‑identical negatives, potentially raising false‑reject rates.

4. Synthetic confusables transfer to real confusables – Training with a single‑edit GraphemeAug set improves AUC on the real‑confusable evaluation set by 54 % relative to the baseline, confirming that synthetic hard negatives can teach the model to reject real‑world acoustic impostors.

5. Asymmetry of real vs synthetic confusables – A model trained on the real‑confusable set performs well on real confusables (AUC ≈ 99.04 %) but poorly on synthetic confusables (AUC ≈ 91.7 %). Conversely, a model trained on synthetic confusables maintains high AUC on both synthetic (98.2 %) and real confusables (99.01 %). This suggests that synthetic generation provides broader phonetic coverage than a modestly sized real confusable corpus.

The authors conclude that systematic grapheme‑level mutation, combined with high‑quality style‑transfer TTS, offers a scalable, language‑agnostic pipeline for producing massive hard‑negative training data. This approach boosts KWS models’ ability to discriminate near‑boundary utterances without sacrificing overall detection accuracy.

Future work is outlined: extending the method to phoneme‑level edits via a grapheme‑to‑phoneme (G2P) front‑end, integrating real user audio for domain adaptation, and evaluating multi‑keyword, multilingual scenarios to verify generalization. The paper positions GraphemeAug as a practical tool for industry‑scale KWS deployments where false activations from rare or evolving utterances are a critical concern.