Universal Sequence Preconditioning

We study the problem of preconditioning in sequential prediction. From the theoretical lens of linear dynamical systems, we show that convolving the target sequence corresponds to applying a polynomial to the hidden transition matrix. Building on this insight, we propose a universal preconditioning method that convolves the target with coefficients from orthogonal polynomials such as Chebyshev or Legendre. We prove that this approach reduces regret for two distinct prediction algorithms and yields the first ever sublinear and hidden-dimension-independent regret bounds (up to logarithmic factors) that hold for systems with marginally table and asymmetric transition matrices. Finally, extensive synthetic and real-world experiments show that this simple preconditioning strategy improves the performance of a diverse range of algorithms, including recurrent neural networks, and generalizes to signals beyond linear dynamical systems.

💡 Research Summary

The paper tackles the longstanding problem of making sequential prediction easier by transforming (pre‑conditioning) the target sequence before learning. While classical differencing (Box‑Jenkins) is a special case of such a transformation, the authors propose a fully general framework: given fixed coefficients (c_0,\dots,c_n) they convolve the observation sequence (y_1,\dots,y_T) with these coefficients, producing a new “pre‑conditioned” sequence (\tilde y_t=\sum_{i=0}^n c_i y_{t-i}).

The key theoretical insight is that, for a linear dynamical system (LDS) defined by
(x_{t+1}=Ax_t+Bu_t,; y_t=Cx_t) (with (D=0)), the pre‑conditioned observation can be rewritten as
\