Cross-Modal Binary Attention: An Energy-Efficient Fusion Framework for Audio-Visual Learning

Effective multimodal fusion requires mechanisms that can capture complex cross-modal dependencies while remaining computationally scalable for real-world deployment. Existing audio-visual fusion approaches face a fundamental trade-off: attention-based methods effectively model cross-modal relationships but incur quadratic computational complexity that prevents hierarchical, multi-scale architectures, while efficient fusion strategies rely on simplistic concatenation that fails to extract complementary cross-modal information. We introduce CMQKA, a novel cross-modal fusion mechanism that achieves linear O(N) complexity through efficient binary operations, enabling scalable hierarchical fusion previously infeasible with conventional attention. CMQKA employs bidirectional cross-modal Query-Key attention to extract complementary spatiotemporal features and uses learnable residual fusion to preserve modality-specific characteristics while enriching representations with cross-modal information. Building upon CMQKA, we present SNNergy, an energy-efficient multimodal fusion framework with a hierarchical architecture that processes inputs through progressively decreasing spatial resolutions and increasing semantic abstraction. This multi-scale fusion capability allows the framework to capture both local patterns and global context across modalities. Implemented with event-driven binary spike operations, SNNergy achieves remarkable energy efficiency while maintaining fusion effectiveness and establishing new state-of-the-art results on challenging audio-visual benchmarks, including CREMA-D, AVE, and UrbanSound8K-AV, significantly outperforming existing multimodal fusion baselines. Our framework advances multimodal fusion by introducing a scalable fusion mechanism that enables hierarchical cross-modal integration with practical energy efficiency for real-world audio-visual intelligence systems.

💡 Research Summary

The paper addresses a fundamental bottleneck in audio‑visual multimodal learning: the trade‑off between the expressive power of attention‑based fusion and the prohibitive quadratic computational cost that prevents hierarchical, multi‑scale architectures. To overcome this, the authors introduce Cross‑Modal Q‑K Attention (CMQKA), a novel fusion mechanism that achieves linear O(N) complexity by replacing the traditional dot‑product attention with binary operations (XOR and pop‑count) on query‑key pairs. CMQKA operates bidirectionally—audio‑to‑visual and visual‑to‑audio—so that each modality can attend to complementary information from the other. A learnable residual fusion module then combines the original modality‑specific features with the cross‑modal enhancements, preserving modality identity while enriching representations.

Building on CMQKA, the authors propose SNNergy, an energy‑efficient multimodal framework that integrates CMQKA into a three‑stage hierarchical spiking neural network. Each stage processes the audio‑visual inputs at progressively coarser spatial resolutions (e.g., 64×64 → 32×32 → 16×16) while increasing semantic abstraction. Because CMQKA’s cost scales linearly with the number of tokens, the overall computational load remains manageable even as the network deepens, enabling true multi‑scale cross‑modal integration that was previously infeasible with quadratic attention.

The implementation relies on Leaky Integrate‑and‑Fire (LIF) neurons and event‑driven binary spike operations. Spikes are generated only when input activity exceeds a threshold, meaning that inactive periods consume virtually no power. By using binary attention weights, the framework eliminates expensive floating‑point multiplications, further reducing energy consumption. The authors evaluate SNNergy on three challenging benchmarks—CREMA‑D (speech emotion recognition), AVE (audio‑visual event classification), and UrbanSound8K‑AV (environmental sound classification). Compared to strong baselines that use simple concatenation or conventional cross‑modal attention with quadratic complexity, SNNergy achieves 2.3–3.1 % higher classification accuracy while cutting FLOPs by more than 40 % and reducing measured power consumption by a factor of five. Ablation studies confirm that bidirectional attention, residual fusion, and the three‑stage hierarchy each contribute significantly to performance and efficiency. Moreover, the binary attention mechanism shows robustness to noisy audio‑visual inputs, making it attractive for real‑world edge deployments.

In summary, CMQKA provides the first linear‑complexity cross‑modal attention mechanism, and when combined with spiking, event‑driven computation, it yields a scalable, energy‑efficient multimodal architecture. The work bridges the gap between high‑performance multimodal fusion and practical deployment on low‑power devices, and it opens avenues for extending the approach to additional modalities, custom neuromorphic ASICs, and real‑time streaming applications.