Neural acoustic multipole splatting for room impulse response synthesis

Room Impulse Response (RIR) prediction at arbitrary receiver positions is essential for practical applications such as spatial audio rendering. We propose Neural Acoustic Multipole Splatting (NAMS), which synthesizes RIRs at unseen receiver positions by learning the positions of neural acoustic multipoles and predicting their emitted signals and directivities using a neural network. Representing sound fields through a combination of multipoles offers sufficient flexibility to express complex acoustic scenes while adhering to physical constraints such as the Helmholtz equation. We also introduce a pruning strategy that starts from a dense splatting of neural acoustic multipoles and progressively eliminates redundant ones during training. Experiments conducted on both real and synthetic datasets indicate that the proposed method surpasses previous approaches on most metrics while maintaining rapid inference. Ablation studies reveal that multipole splatting with pruning achieves better performance than the monopole model with just 20% of the poles.

💡 Research Summary

This paper introduces Neural Acoustic Multipole Splatting (NAMS), a novel framework for synthesizing room impulse responses (RIRs) at arbitrary listener positions. Unlike prior methods that either rely on purely physics‑based models (modal expansion, equivalent source, plane‑wave expansion) or pure data‑driven neural networks, NAMS combines the physical rigor of the Helmholtz equation with the expressive power of deep learning. The core idea is to represent the acoustic field as a superposition of acoustic multipoles—sources that emit a short‑duration signal and possess a frequency‑dependent directional pattern expressed via spherical harmonics. Each multipole p is characterized by three learnable entities: its spatial location xₚ, its emitted time‑domain signal sₚ(t), and a set of spherical‑harmonic coefficients bₙₘ,ₚ(t) that define its directivity Dₚ(f, xᵣ). Two separate MLPs predict these quantities: a “signal branch” takes only the multipole position (ensuring source signals are independent of the receiver) and outputs sₚ(t); a “directivity branch” receives the relative vector (xᵣ – xₚ), encodes it, and produces the time‑domain coefficients that are transformed into frequency‑domain directivity via Eq. (1). The RIR in the frequency domain is then assembled as H(f, xᵣ)=∑ₚ Sₚ(f)·e^{-j2πf rₚ/c}·(1/rₚ)·Dₚ(f, xᵣ), where rₚ=‖xᵣ–xₚ‖. This formulation automatically satisfies the Helmholtz equation because each term is a valid solution.

Training minimizes a composite loss comprising spectral, amplitude, phase, time‑domain, multi‑resolution STFT, and energy‑decay components, following the loss design of A‑VR. To stabilize learning, the directivity energy is normalized to unity, and the signal energy is constrained via an L2 norm. The model is fully differentiable, allowing joint optimization of multipole positions and MLP weights by back‑propagation.

A key contribution is the pruning strategy. The authors start with a dense set of 1,089 multipoles placed on concentric spheres (radii 1–34 m, 32 points per sphere) plus one at the source. Every 20 epochs after an initial 100‑epoch warm‑up, they evaluate the energy of each sₚ(t). Multipoles whose energy falls below 50 % of the global median are removed. This iterative “dense‑then‑prune” process yields a compact model with roughly 225–276 active multipoles, dramatically reducing computational load while preventing over‑fitting.

Experiments were conducted on the real‑world MeshRIR dataset and two synthetic rooms (apartments 566 and 716) generated with the Treble simulator. All RIRs were resampled to 24 kHz and truncated to 0.1 s. The network uses sinusoidal positional encodings (10 frequencies) and three‑layer MLPs with 512 hidden units for both branches. The signal output length is 3 ms; directivity outputs include spherical‑harmonic coefficients up to third order (16 channels). Training runs for 300 epochs on an RTX A6000 GPU with Adam and a cosine learning‑rate schedule (1e‑3 → 1e‑4).

Performance is evaluated using amplitude error, envelope error (%), T₆₀ error (%), clarity C₅₀ error (dB), and early decay time (EDT) error (ms). On MeshRIR, NAMS achieves amplitude error 0.11, envelope error 1.21 %, T₆₀ error 2.0 %, C₅₀ error 0.34 dB, and EDT error 9.8 ms—substantially better than NAF (0.57, 1.98 %, 3.5 %, 0.88 dB, 31 ms) and A‑VR (0.28, 1.44 %, 2.9 %, 0.66 dB, 19.4 ms). Similar trends appear on the synthetic rooms, with NAMS consistently outperforming baselines on most metrics. Spatial magnitude visualizations reveal smoother, less jittery fields for NAMS. In terms of resources, NAMS uses 1.8 M parameters and 2.1–2.2 ms inference time, comparable to NAF (2.7 M, ~2 ms) but far more efficient than A‑VR, which requires 57 M parameters (including a hash‑grid encoder) and 205 k sample points, leading to higher memory and compute demands.

Ablation studies compare monopole (N=0) versus multipole (N=3) configurations and dense versus sparse initializations. Multipole models consistently outperform monopoles, and pruning reduces the number of poles to about 20 % of the original dense set without sacrificing accuracy, confirming the effectiveness of the adaptive sparsity scheme.

In summary, NAMS demonstrates that a physically grounded multipole representation, learned end‑to‑end with neural networks and refined through energy‑based pruning, can deliver high‑fidelity, low‑latency RIR synthesis. This approach bridges the gap between accurate acoustic modeling and real‑time applicability, opening pathways for immersive audio in AR/VR, gaming, and interactive sound design where rapid generation of spatially varying RIRs is essential.