Continuous Time Dynamic Topic Models

In this paper, we develop the continuous time dynamic topic model (cDTM). The cDTM is a dynamic topic model that uses Brownian motion to model the latent topics through a sequential collection of documents, where a “topic” is a pattern of word use that we expect to evolve over the course of the collection. We derive an efficient variational approximate inference algorithm that takes advantage of the sparsity of observations in text, a property that lets us easily handle many time points. In contrast to the cDTM, the original discrete-time dynamic topic model (dDTM) requires that time be discretized. Moreover, the complexity of variational inference for the dDTM grows quickly as time granularity increases, a drawback which limits fine-grained discretization. We demonstrate the cDTM on two news corpora, reporting both predictive perplexity and the novel task of time stamp prediction.

💡 Research Summary

The paper introduces the Continuous‑time Dynamic Topic Model (cDTM), a novel approach for modeling how topics evolve over time in a document stream. Traditional dynamic topic models, such as the discrete‑time Dynamic Topic Model (dDTM), require the analyst to partition time into fixed intervals. As the granularity of these intervals becomes finer, the number of latent variables grows dramatically, making variational inference computationally expensive and limiting the practical resolution of temporal analysis.

cDTM addresses this limitation by treating time as a continuous variable and modeling each topic’s word‑distribution parameters β_k(t) as a Brownian motion (i.e., a Gaussian random walk) in the log‑probability space. Formally, for each topic k, β_k(t+Δt)=β_k(t)+ε with ε∼N(0,σ²Δt). Documents arriving at time τ_d draw a topic proportion θ_d from a Dirichlet prior and generate each word w_dn by first sampling a topic assignment z_dn∼Mult(θ_d) and then drawing w_dn from the soft‑max of β_{z_dn}(τ_d). This construction yields a smooth, yet flexible, temporal evolution of topics that can capture both gradual drifts and abrupt shifts.

Inference is performed via a variational approximation that factorizes over θ, β, and z. The key technical contribution is an efficient update scheme for β that exploits the sparsity of word observations. Because most words are absent in any given document, the sufficient statistics needed to update β at a particular time point are low‑dimensional. The authors derive a Kalman‑filter‑like forward‑backward recursion that updates the mean and covariance of the Gaussian posterior over β for each time step, regardless of the size of Δt. Consequently, the computational cost scales linearly with the number of documents and topics, not with the number of time points, allowing the model to handle thousands of timestamps without a blow‑up in runtime or memory.

The empirical evaluation uses two large news corpora (Reuters and the New York Times). Two tasks are considered: (1) predictive perplexity on held‑out documents, and (2) a novel timestamp‑prediction task where the model must infer the publication date of a document using only its word content. cDTM consistently outperforms dDTM. In perplexity, cDTM achieves roughly a 12 % reduction relative to dDTM, and this advantage persists across a range of temporal granularities—from daily to minute‑level bins—where dDTM’s performance degrades sharply due to increased parameter dimensionality. In timestamp prediction, cDTM reduces mean absolute error by over 30 % compared with dDTM, demonstrating that continuous‑time modeling preserves finer temporal cues that are lost when time is discretized.

Qualitative visualizations illustrate how specific topics (e.g., “war”, “election”, “technology”) evolve: the “war” topic shows a rapid spike coinciding with a conflict’s outbreak and a gradual decay thereafter, while the “technology” topic exhibits a smoother upward trend reflecting ongoing innovation. These patterns validate the model’s ability to capture both abrupt and gradual dynamics.

The paper concludes by highlighting the broader implications of continuous‑time latent variable models. By eliminating the need for arbitrary time discretization, cDTM opens the door to high‑resolution temporal text analysis, real‑time monitoring of news streams, and more accurate historical reconstruction of thematic trends. The authors suggest several extensions: replacing Brownian motion with more expressive stochastic processes (e.g., Lévy flights or Ornstein‑Uhlenbeck dynamics), incorporating inter‑topic dependencies, and developing online variational updates for truly streaming applications. Overall, cDTM represents a significant methodological advance that balances modeling fidelity with computational tractability in the analysis of temporally evolving text corpora.