Frequency Domain Multi-channel Acoustic Modeling for Distant Speech Recognition

Conventional far-field automatic speech recognition (ASR) systems typically employ microphone array techniques for speech enhancement in order to improve robustness against noise or reverberation. However, such speech enhancement techniques do not always yield ASR accuracy improvement because the optimization criterion for speech enhancement is not directly relevant to the ASR objective. In this work, we develop new acoustic modeling techniques that optimize spatial filtering and long short-term memory (LSTM) layers from multi-channel (MC) input based on an ASR criterion directly. In contrast to conventional methods, we incorporate array processing knowledge into the acoustic model. Moreover, we initialize the network with beamformers’ coefficients. We investigate effects of such MC neural networks through ASR experiments on the real-world far-field data where users are interacting with an ASR system in uncontrolled acoustic environments. We show that our MC acoustic model can reduce a word error rate (WER) by16.5% compared to a single channel ASR system with the traditional log-mel filter bank energy (LFBE) feature on average. Our result also shows that our network with the spatial filtering layer on two-channel input achieves a relative WER reduction of9.5% compared to conventional beamforming with seven microphones.

💡 Research Summary

The paper addresses a fundamental limitation in conventional far‑field automatic speech recognition (ASR) pipelines: the separation of microphone‑array beamforming from acoustic modeling. Traditional systems first apply a beamformer—often a fixed or adaptive super‑directive (SD) beamformer—designed to improve signal‑to‑noise ratio (SNR) or reduce reverberation, and then feed the enhanced single‑channel waveform into a feature extractor (e.g., log‑mel filter‑bank energies, LFBE) and a neural acoustic model (typically LSTM‑based). Because the beamformer is optimized for signal‑processing criteria rather than the ASR objective, improvements in SNR do not always translate into lower word error rates (WER).

To close this gap, the authors propose fully learnable multi‑channel (MC) acoustic models that operate directly on frequency‑domain representations of the raw microphone signals. The core idea is to embed spatial filtering—traditionally performed by a beamformer—inside a neural network that is jointly trained with the acoustic classifier using the cross‑entropy loss that directly reflects ASR performance. Three distinct MC network architectures are explored:

1. Complex Affine Transform (CAT) – a straightforward complex‑valued linear projection followed by a complex‑squared non‑linearity. This mimics the complex linear projection models previously proposed but adds a bias term to increase flexibility.

2. Deterministic Spatial Filtering (DSF) – the first layer is initialized with SD beamformer weight vectors for a set of discrete look‑directions. After applying these filters, the network computes the power (sum of squares of real and imaginary parts) for each direction and selects the direction with maximum power via a max‑pooling operation. This implements a “best‑energy” beamformer selection inside the network while still allowing the weights to be fine‑tuned during training. An additional degree of freedom is introduced by allowing the spatial filtering layer to interact across frequencies, which mitigates irreversible selection errors.

3. Elastic Spatial Filtering (ESF) – extends DSF by combining the power outputs of multiple beamformers rather than picking a single winner. The outputs are passed through a frequency‑independent affine transform, a ReLU, and a logarithm, effectively learning a weighted sum of beamformer responses. This architecture preserves the computational efficiency of frequency‑independent processing while providing robustness to beamformer selection mistakes.

All three architectures accept as input the normalized complex DFT coefficients of each microphone channel (127 dimensions after discarding DC and Nyquist bins). The network proceeds with a feature‑extraction DNN that mimics LFBE (affine transform initialized with mel‑filter weights, ReLU, log) and then a standard five‑layer LSTM acoustic model (768 cells per layer) followed by an affine transform and softmax over senone targets. Training follows a staged approach: (i) pre‑train the LSTM classifier on single‑channel LFBE, (ii) train the feature‑extraction DNN on single‑channel DFT features, and (iii) jointly fine‑tune the entire MC network on multi‑channel DFT inputs. The Adam optimizer and cross‑entropy loss are used throughout.

Experiments are conducted on a large in‑house dataset comprising over 1,100 hours of real user interactions captured with a seven‑microphone circular array (six microphones on a 72 mm diameter ring plus a central mic). The test set contains 50 hours of speech from speakers not seen during training, recorded in uncontrolled acoustic environments (varying noise, reverberation, and speaker movement). Baseline systems include a conventional SD beamformer with diagonal loading and a single‑channel LFBE‑LSTM model. For the MC models, two‑channel configurations (diagonal microphone pair) are also evaluated to assess performance with fewer sensors.

Results are reported as relative WER reduction (WERR) compared to the baseline. The ESF model using all seven channels achieves an average 16.5 % WERR over the LFBE baseline. Remarkably, the DSF model with only two channels attains a 9.5 % relative WERR improvement over the conventional seven‑channel SD beamformer, demonstrating that the learned spatial filtering can outperform hand‑crafted beamforming even with fewer microphones. The CAT model yields modest gains, indicating that simply learning a complex linear projection without explicit spatial constraints is insufficient. Adaptive beamforming approaches were tested but degraded performance due to unreliable voice‑activity detection and speaker localization in the real‑world data, reinforcing the advantage of the proposed end‑to‑end MC networks.

Key contributions of the work are: (1) a unified, fully differentiable framework that merges array signal processing and acoustic modeling in the frequency domain; (2) the use of beamformer weights as a principled initialization that accelerates convergence and embeds domain knowledge; (3) extensive validation on large‑scale, real‑world far‑field speech data showing consistent WER reductions; and (4) demonstration that even minimal microphone configurations can surpass traditional multi‑mic beamforming when the spatial filter is learned jointly with the recognizer.

The paper suggests future directions such as investigating data‑efficient training (e.g., semi‑supervised or transfer learning) to reduce the large data requirement, exploring hardware‑friendly implementations for low‑latency streaming ASR, and extending the approach to irregular or ad‑hoc microphone arrays where explicit array geometry may be unknown. Overall, the study provides compelling evidence that integrating spatial filtering directly into the acoustic model, optimized for the ASR loss, yields superior robustness for distant speech recognition in realistic environments.