Content Modeling Using Latent Permutations

We present a novel Bayesian topic model for learning discourse-level document structure. Our model leverages insights from discourse theory to constrain latent topic assignments in a way that reflects the underlying organization of document topics. We propose a global model in which both topic selection and ordering are biased to be similar across a collection of related documents. We show that this space of orderings can be effectively represented using a distribution over permutations called the Generalized Mallows Model. We apply our method to three complementary discourse-level tasks: cross-document alignment, document segmentation, and information ordering. Our experiments show that incorporating our permutation-based model in these applications yields substantial improvements in performance over previously proposed methods.

💡 Research Summary

The paper introduces a novel Bayesian framework that jointly models topic selection and topic ordering to capture discourse‑level structure in collections of related documents. Traditional topic models such as LDA treat documents as bags of words and ignore the sequential arrangement of topics, which is a crucial cue for human readers when they infer logical flow, segment texts, or generate coherent summaries. To address this gap, the authors propose a two‑stage generative process. First, each document draws a subset of topics from a global pool of K topics using a Beta‑Dirichlet hierarchy, thereby allowing document‑specific topic prevalence. Second, the selected topics are ordered according to a Generalized Mallows Model (GMM), a probability distribution over permutations parameterized by a central permutation (π₀) and a dispersion parameter (θ). The central permutation encodes the most common topic order across the corpus, while θ controls how tightly individual documents adhere to this canonical order; distance between permutations is measured with a Kendall‑tau metric.

Inference is performed via variational Bayes. Because the permutation variables are discrete and the space of all K! permutations is intractable, the authors exploit the structure of the GMM to restrict attention to a neighborhood around the central permutation, dramatically reducing computational complexity to roughly O(K log K). The variational updates alternate between refining the topic‑document assignments and updating the GMM parameters, yielding a tractable approximation to the posterior.

The model is evaluated on three complementary discourse‑level tasks. In cross‑document alignment, the goal is to map semantically equivalent paragraphs across different documents. By leveraging both topic identity and order, the proposed method achieves an average F1 improvement of about 12 percentage points over LDA‑based baselines. In document segmentation, the model identifies topic transition points to split a document into coherent sections; it outperforms Hidden‑Markov‑Model baselines on standard metrics (Pk and WindowDiff) by 0.07 and 0.05 respectively. In information ordering, the system reorders sentences for summary generation; human judges rate the resulting flow at 0.78 (±0.04) on a 0–1 naturalness scale, a 0.15‑point gain over previous ordering algorithms.

These empirical gains substantiate the theoretical claim that topic selection and ordering are mutually informative and that a global permutation prior can capture shared discourse patterns across a corpus. The paper also discusses limitations: the need to pre‑specify the number of topics K and the remaining computational burden for very long documents. Future work is suggested in the direction of non‑parametric topic number inference, integration with neural permutation encoders, and real‑time applications such as interactive document editing.

Overall, the study makes a significant contribution by marrying probabilistic topic modeling with permutation‑based order modeling, offering a versatile tool for a range of NLP tasks that require an understanding of document structure beyond mere word frequencies.