GE2E-KWS: Generalized End-to-End Training and Evaluation for Zero-shot Keyword Spotting

We propose GE2E-KWS – a generalized end-to-end training and evaluation framework for customized keyword spotting. Specifically, enrollment utterances are separated and grouped by keywords from the training batch and their embedding centroids are compared to all other test utterance embeddings to compute the loss. This simulates runtime enrollment and verification stages, and improves convergence stability and training speed by optimizing matrix operations compared to SOTA triplet loss approaches. To benchmark different models reliably, we propose an evaluation process that mimics the production environment and compute metrics that directly measure keyword matching accuracy. Trained with GE2E loss, our 419KB quantized conformer model beats a 7.5GB ASR encoder by 23.6% relative AUC, and beats a same size triplet loss model by 60.7% AUC. Our KWS models are natively streamable with low memory footprints, and designed to continuously run on-device with no retraining needed for new keywords (zero-shot).

💡 Research Summary

The paper introduces GE2E‑KWS, a unified framework for zero‑shot customized keyword spotting that eliminates the need for retraining when new keywords are added. The core idea is to adopt the Generalized End‑to‑End (GE2E) loss, originally used for speaker verification, and apply it to keyword matching. During training, each batch contains X different keywords, each with Y utterances. For every keyword, Y/2 utterances are treated as enrollment and the remaining Y/2 as test. The enrollment embeddings are averaged to form a centroid c_i. The loss encourages high cosine similarity between c_i and its own test embeddings (positives) while pushing down similarity with test embeddings of other keywords (negatives). This batch‑wise centroid‑to‑all‑test formulation reduces sampling variance compared to traditional triplet loss, stabilizes convergence, and enables efficient matrix‑based computation.

The authors evaluate two families of acoustic encoders: conventional LSTM stacks and state‑of‑the‑art Conformer networks. Conformer models are tuned in depth, attention heads, and embedding size to explore the size‑accuracy trade‑off. After training, the best Conformer (2.8 MB) is quantized with TensorFlow Lite dynamic range quantization to 419 KB, preserving most of the original accuracy while enabling on‑device streaming inference.

For evaluation, the authors propose a realistic protocol that mimics production deployment. Using the Speech Commands dataset (35 keywords, ~11 k utterances), they randomly select 10 enrollment utterances per keyword, compute a centroid, and compare it against all remaining test utterances via cosine similarity. They report detection error trade‑off (DET) curves, area under the DET curve (AUC), and equal error rate (EER), both in clean conditions and with multi‑talker noise (3‑15 dB). This metric suite directly measures keyword‑matching quality rather than indirect classification accuracy.

Results show that the 419 KB quantized Conformer surpasses a 7.5 GB pre‑trained ASR encoder by 23.6 % relative AUC and outperforms a same‑architecture triplet‑loss baseline by 60.7 % relative AUC. The model also achieves substantially lower EER across all keywords. Importantly, the model is fully streamable, has a tiny memory footprint, and can run continuously on edge devices without any additional training for new keywords—fulfilling the zero‑shot requirement.

In summary, GE2E‑KWS delivers a comprehensive solution: a loss function that mirrors the enrollment‑verification workflow, a scalable architecture (especially Conformer) that can be aggressively quantized, and an evaluation methodology that aligns with real‑world usage. The work paves the way for on‑device, user‑personalized voice assistants that can instantly recognize arbitrary custom wake‑words while respecting strict latency and memory constraints. Future directions include multilingual support, text‑to‑speech enrollment pipelines, and on‑device adaptation to further improve robustness.