Transformers as Implicit State Estimators: In-Context Learning in Dynamical Systems

Predicting the behavior of a dynamical system from noisy observations of its past outputs is a classical problem encountered across engineering and science. For linear systems with Gaussian inputs, the Kalman filter – the best linear minimum mean-square error estimator of the state trajectory – is optimal in the Bayesian sense. For nonlinear systems, Bayesian filtering is typically approached using suboptimal heuristics such as the Extended Kalman Filter (EKF), or numerical methods such as particle filtering (PF). In this work, we show that transformers, employed in an in-context learning (ICL) setting, can implicitly infer hidden states in order to predict the outputs of a wide family of dynamical systems, without test-time gradient updates or explicit knowledge of the system model. Specifically, when provided with a short context of past input-output pairs and, optionally, system parameters, a frozen transformer accurately predicts the current output. In linear-Gaussian regimes, its predictions closely match those of the Kalman filter; in nonlinear regimes, its performance approaches that of EKF and PF. Moreover, prediction accuracy degrades gracefully when key parameters, such as the state-transition matrix, are withheld from the context, demonstrating robustness and implicit parameter inference. These findings suggest that transformer in-context learning provides a flexible, non-parametric alternative for output prediction in dynamical systems, grounded in implicit latent-state estimation.

💡 Research Summary

The paper “Transformers as Implicit State Estimators: In‑Context Learning in Dynamical Systems” investigates whether a frozen transformer, trained only on synthetic trajectories from randomly sampled dynamical systems, can perform state‑estimation‑like inference when prompted with a short context of past input‑output pairs. The authors frame the problem in the classic state‑space setting, contrasting linear‑Gaussian systems where the Kalman filter is optimal with nonlinear systems that typically require the Extended Kalman Filter (EKF) or particle filters (PF).

First, they provide a constructive proof that the Kalman filter’s prediction and update equations can be expressed using operations native to a transformer: soft‑max attention can emulate the computation of the Kalman gain, while the feed‑forward network can implement the covariance update. By choosing appropriate key, query, and value matrices, a single attention head can reproduce the linear algebraic steps of the optimal linear estimator.

Empirically, the authors pre‑train a GPT‑2‑style decoder on millions of trajectories generated from random linear systems (varying transition matrix F, observation matrix H, and noise covariances Q, R). At test time, they feed the model a prompt consisting of a sequence of past inputs uₜ, outputs yₜ, and optionally the system parameters, and ask it to predict the next output. When the context length L is at least as large as the state dimension n and the model contains ≥100 M parameters, the mean‑squared‑prediction‑difference (MSPD) matches the Kalman filter to within 0.1 %. Smaller models or shorter contexts fall back to behaviours akin to stochastic gradient descent or ridge regression, confirming a scale‑dependent transition from simple regression to full‑state inference.

For nonlinear dynamics, the authors test a maneuvering‑target tracking scenario with unknown turning rate and highly nonlinear measurement functions. The frozen transformer, without any gradient updates, attains prediction errors comparable to EKF and PF, sometimes outperforming them by a few percent. Notably, when key parameters such as the state‑transition matrix are omitted from the prompt, performance degrades gracefully and resembles that of a dual‑Kalman filter that must estimate the missing matrix online. This demonstrates that the transformer can implicitly infer missing system parameters from the observed data stream.

The paper also explores robustness to limited context, model capacity, and noise levels. It shows that with very short contexts the transformer behaves like an online learner (SGD), while with long contexts it effectively performs batch‑style least‑squares estimation, implicitly reconstructing latent states. The authors discuss that this behaviour emerges from the transformer’s ability to store and retrieve information across tokens via self‑attention, effectively acting as a differentiable memory that aggregates past observations.

Limitations are acknowledged: experiments are confined to synthetic data, and real‑world deployment (e.g., robotics, power‑grid forecasting) would require handling non‑stationary noise, control inputs, and strict latency constraints. The authors suggest future work on explicit latent‑state decoding, integration with control loops, and model compression for real‑time use.

In summary, the study provides both theoretical and empirical evidence that large pretrained transformers can serve as non‑parametric, in‑context estimators for dynamical systems, matching or surpassing classical Bayesian filters without any test‑time adaptation. This opens a new avenue where language‑model architectures are repurposed for signal‑processing and control tasks, offering a flexible alternative that leverages the massive pretraining of modern transformers while retaining the principled performance of optimal filters.