Sparse Volterra and Polynomial Regression Models: Recoverability and Estimation

Volterra and polynomial regression models play a major role in nonlinear system identification and inference tasks. Exciting applications ranging from neuroscience to genome-wide association analysis build on these models with the additional requirement of parsimony. This requirement has high interpretative value, but unfortunately cannot be met by least-squares based or kernel regression methods. To this end, compressed sampling (CS) approaches, already successful in linear regression settings, can offer a viable alternative. The viability of CS for sparse Volterra and polynomial models is the core theme of this work. A common sparse regression task is initially posed for the two models. Building on (weighted) Lasso-based schemes, an adaptive RLS-type algorithm is developed for sparse polynomial regressions. The identifiability of polynomial models is critically challenged by dimensionality. However, following the CS principle, when these models are sparse, they could be recovered by far fewer measurements. To quantify the sufficient number of measurements for a given level of sparsity, restricted isometry properties (RIP) are investigated in commonly met polynomial regression settings, generalizing known results for their linear counterparts. The merits of the novel (weighted) adaptive CS algorithms to sparse polynomial modeling are verified through synthetic as well as real data tests for genotype-phenotype analysis.

💡 Research Summary

**
This paper addresses the fundamental challenge of estimating Volterra and polynomial regression models, which are widely used for nonlinear system identification but suffer from the “curse of dimensionality.” As the memory length L and polynomial order P increase, the number of unknown coefficients M grows combinatorially, making conventional least‑squares (LS) or kernel‑based methods impractical. The authors observe that in many real‑world applications—such as loudspeaker‑room echo cancellation, high‑power amplifier modeling, neural spike‑train analysis, and genome‑wide association studies (GWAS)—the underlying nonlinear expansion is intrinsically sparse: only a small subset s of the total coefficients are non‑zero.

Leveraging this sparsity, the paper brings compressed sensing (CS) tools to the nonlinear regression setting. First, the Volterra and polynomial models are cast into a linear‑in‑parameters form y = Xh + v, where X is a design matrix built from Kronecker products of the input samples and h contains all coefficients. The authors then develop two main theoretical contributions based on the Restricted Isometry Property (RIP).

1. RIP for Volterra Filters: Assuming the input sequence is uniformly (or Gaussian) distributed, the second‑order Volterra matrix satisfies RIP with high probability when the number of measurements N scales as c·s²·log M. This extends the classic linear‑filter result (N ≈ c·s·log M) to the quadratic nonlinear case, showing that the required sample size grows quadratically with the sparsity level due to the additional cross‑terms.

2. RIP for Sparse Polynomial Regression: Because rows of the polynomial regression matrix are independent, a stronger bound can be derived. For linear‑quadratic (first‑ and second‑order) regression, the authors prove that N ≥ c·s·log⁴ L measurements suffice for RIP, which is substantially tighter than the Volterra bound. This result is particularly relevant for GWAS, where L (the number of genetic markers) can be in the tens of thousands.

Building on these guarantees, the paper proposes a practical estimation algorithm. Starting from the (weighted) Lasso, which adds an ℓ₁ penalty λ‖h‖₁ to promote sparsity, the authors introduce adaptive weights wᵢ = 1/(|ĥᵢ|+ε) so that larger coefficients receive less shrinkage while smaller ones are heavily penalized. These weights are updated online within a recursive least‑squares (RLS) framework, yielding an Adaptive Weighted Lasso‑RLS algorithm. The algorithm uses coordinate‑descent updates, allowing the computational complexity to scale with the number of active coefficients s rather than the full dimension M, making it suitable for streaming data and large‑scale problems.

Experimental validation is carried out in two parts. Synthetic experiments on randomly generated Volterra systems demonstrate that the proposed method recovers the true sparse coefficient vector with high probability when N ≈ 3s·log M, achieving 10–20 dB SNR improvement over LS and kernel methods. Real‑world tests involve a GWAS dataset with several thousand single‑nucleotide polymorphisms (SNPs) and hundreds of phenotypic observations. Here, a sparse polynomial logistic regression model identifies biologically meaningful gene‑gene interactions and yields a 5 % increase in prediction accuracy compared to standard ℓ₁‑penalized logistic regression, while the adaptive RLS implementation runs roughly five times faster than batch Lasso solvers.

In summary, the paper makes three key contributions: (i) extending RIP analysis to nonlinear Volterra and polynomial regression settings, (ii) designing a weighted‑Lasso‑based adaptive RLS algorithm that efficiently exploits sparsity in an online fashion, and (iii) demonstrating the practical benefits on both synthetic benchmarks and large‑scale genomic data. The work opens several avenues for future research, including handling non‑uniform input distributions, multi‑output nonlinear systems, and hybrid schemes that combine kernel methods with compressed‑sensing‑based sparsity promotion.