LatentTrack: Sequential Weight Generation via Latent Filtering

We introduce LatentTrack (LT), a sequential neural architecture for online probabilistic prediction under nonstationary dynamics. LT performs causal Bayesian filtering in a low-dimensional latent space and uses a lightweight hypernetwork to generate predictive model parameters at each time step, enabling constant-time online adaptation without per-step gradient updates. At each time step, a learned latent model predicts the next latent distribution, which is updated via amortized inference using new observations, yielding a predict–generate–update filtering framework in function space. The formulation supports both structured (Markovian) and unstructured latent dynamics within a unified objective, while Monte Carlo inference over latent trajectories produces calibrated predictive mixtures with fixed per-step cost. Evaluated on long-horizon online regression using the Jena Climate benchmark, LT consistently achieves lower negative log-likelihood and mean squared error than stateful sequential and static uncertainty-aware baselines, with competitive calibration, demonstrating that latent-conditioned function evolution is an effective alternative to traditional latent-state modeling under distribution shift.

💡 Research Summary

LatentTrack (LT) introduces a novel “function‑space filtering” paradigm for online probabilistic prediction under non‑stationary dynamics. Instead of filtering over observations or latent hidden states, LT maintains a low‑dimensional latent variable zₜ that directly generates the parameters θₜ of a predictive model via a lightweight hypernetwork g_η. At each time step the system follows a predict‑generate‑update cycle: a learned prior predicts the next latent distribution, amortized inference updates the posterior over zₜ using the newly observed data, and the hypernetwork maps the sampled latent state to a fresh set of model weights. This design concentrates expressive capacity in the weight‑generation stage while keeping the latent dynamics simple, a hypothesis the authors validate empirically.

The training objective is a variational filtering ELBO. The generic form (Eq. 2) bounds the log‑likelihood of the current observation by an expected log‑likelihood term under the amortized posterior and a KL divergence between the posterior q_ψ(zₜ|D₁:ₜ) and a one‑step prior p_ϕ(zₜ|D₁:ₜ₋₁). Two ELBO variants are explored: (1) a “structured” KL (Eq. 3) that conditions the prior on the previous latent state zₜ₋₁, encouraging temporal coherence and reducing latent drift, and (2) an “unstructured” KL that compares directly to the marginal prior. The structured version, called LT‑Structured, consistently yields more stable rankings and lower negative log‑likelihood (NLL) in experiments, while the unstructured version (LT‑Unstructured) serves as a baseline.

Latent inference is amortized through a recurrent summarizer hₜ implemented as a GRU. Each incoming supervised pair Dₜ = (xₜ, yₜ) is encoded, combined with the previous summary, and used to produce the parameters of the prior distribution p_ϕ(zₜ|hₜ₋₁). During inference, K samples of zₜ are drawn from this prior (typically K = 100), passed through the hypernetwork to obtain K weight sets θₖₜ, and combined into a predictive mixture p̂(yₜ|xₜ) by averaging. For Gaussian heads this yields a mean equal to the average of the K predictive means and a variance that decomposes into aleatoric (average of per‑sample variances) and epistemic (variance of the predictive means) components. Crucially, test‑time adaptation requires no gradient computation; the model simply samples a new latent state and generates new weights, achieving O(1) per‑step computational cost.

The authors evaluate LT on the Jena Climate dataset, a multivariate hourly time‑series with strong seasonality and drift. After down‑sampling to a 6‑hour resolution, the task is to predict temperature six steps ahead (36 hours) in a strictly causal streaming setting: predictions at time t use data up to t − 1, after which the latent state is updated with the newly observed pair. The first 70 % of the sequence is used for training, the remaining 30 % for evaluation, which includes several regime changes. Baselines include stateful sequential models (Variational RNNs, Deep State‑Space Models) and static uncertainty‑aware models (Monte‑Carlo Dropout, Deep Ensembles, Bayes‑by‑Backprop). All methods are matched in total parameter count (~20 k) and test‑time sampling budget to ensure a fair comparison.

Results show that LT‑Structured achieves the lowest trimmed‑mean NLL (6.29) and the highest proportion of time steps where it ranks first (≈58 % of steps). Its trimmed‑mean MSE (≈10 k) also outperforms all baselines, with a first‑place rank in about 51 % of steps. The unstructured variant performs worse, confirming the benefit of the structured KL. State‑based models such as VRNN and DSSM lag behind both in NLL and MSE, while static ensembles, despite having multiple members, cannot maintain temporal consistency and suffer higher catastrophic‑failure rates (instances where NLL exceeds 10⁶). Calibration diagnostics (PIT histograms, reliability diagrams) reported in the appendix indicate that LT’s predictive mixtures are well‑calibrated, with epistemic uncertainty growing during distribution shifts.

The paper’s contributions are fourfold: (1) a new function‑space filtering framework, (2) a predict‑generate‑update ELBO that works for both Markovian and non‑Markovian latent dynamics, (3) a capacity‑allocation strategy that places most expressive power in the hypernetwork rather than in latent inference, and (4) a demonstration of constant‑time online adaptation with superior predictive performance and calibrated uncertainty.

Limitations include the fixed latent dimensionality (d = 8) which may be insufficient for more complex non‑stationarities, the use of a relatively simple linear‑Gaussian hypernetwork that could be replaced by richer nonlinear mappings, and the assumption that covariates xₜ are observed without modeling their dynamics. Moreover, increasing the Monte‑Carlo sample count K improves uncertainty estimates but raises computational load, which may be problematic for strict real‑time constraints.

Future work could explore larger or adaptive latent spaces, more expressive hypernetworks (e.g., transformer‑based generators), incorporation of covariate dynamics, and application to other domains such as finance, healthcare, or reinforcement‑learning policy adaptation. The function‑space filtering idea also opens avenues for meta‑learning and continual learning where the model’s function evolves rather than its parameters staying static.

In summary, LatentTrack presents a compelling alternative to traditional latent‑state modeling: by generating the predictive function itself from a filtered latent variable, it achieves constant‑time online adaptation, higher accuracy, and reliable uncertainty quantification under distribution shift, marking a significant step forward for streaming machine‑learning systems.