A Poisson convolution model for characterizing topical content with word frequency and exclusivity

An ongoing challenge in the analysis of document collections is how to summarize content in terms of a set of inferred themes that can be interpreted substantively in terms of topics. The current practice of parametrizing the themes in terms of most frequent words limits interpretability by ignoring the differential use of words across topics. We argue that words that are both common and exclusive to a theme are more effective at characterizing topical content. We consider a setting where professional editors have annotated documents to a collection of topic categories, organized into a tree, in which leaf-nodes correspond to the most specific topics. Each document is annotated to multiple categories, at different levels of the tree. We introduce a hierarchical Poisson convolution model to analyze annotated documents in this setting. The model leverages the structure among categories defined by professional editors to infer a clear semantic description for each topic in terms of words that are both frequent and exclusive. We carry out a large randomized experiment on Amazon Turk to demonstrate that topic summaries based on the FREX score are more interpretable than currently established frequency based summaries, and that the proposed model produces more efficient estimates of exclusivity than with currently models. We also develop a parallelized Hamiltonian Monte Carlo sampler that allows the inference to scale to millions of documents.

💡 Research Summary

The paper tackles the longstanding problem of summarizing large document collections in terms of interpretable topics. Traditional topic models, such as Latent Dirichlet Allocation (LDA) and its extensions, typically describe a topic by the most frequent words, ignoring how uniquely those words are used by a particular topic compared to others. This reliance on raw frequency often leads to ambiguous or overlapping topic descriptions, especially when the underlying taxonomy is hierarchical and documents are annotated with multiple labels at different levels of granularity.

To address these shortcomings, the authors introduce a Hierarchical Poisson Convolution (HPC) model that explicitly incorporates a professional‑curated taxonomy of categories organized as a tree. Each node in the tree represents a topic, and each topic is associated with a vector of Poisson rate parameters λ_{t,w} governing the expected count of word w in documents labeled with topic t. The hierarchical structure is encoded by treating the λ of a parent node as a Bayesian prior for its children, thereby allowing information to flow from coarse to fine topics and mitigating data sparsity at lower levels. Documents may carry multiple labels, and the observed word counts are modeled as a Poisson mixture over the λ vectors of all assigned topics.

Inference is performed using Hamiltonian Monte Carlo (HMC), a gradient‑based Markov chain Monte Carlo technique that efficiently explores high‑dimensional posterior spaces. The authors develop a parallelized HMC implementation that leverages GPU acceleration to compute log‑likelihoods and gradients for millions of documents simultaneously. Adaptive step‑size and mass‑matrix tuning further improve convergence, enabling the model to scale to corpora with millions of documents and thousands of topics.

A central contribution is the definition of the FREX (Frequency‑Exclusivity) score, which ranks words for a given topic by combining two complementary criteria: (1) frequency, measured as the normalized λ_{t,w} relative to the overall average across topics, and (2) exclusivity, measured as the degree to which λ_{t,w} exceeds the average λ for the same word in all other topics. The two components are blended with a user‑specified weight α, allowing practitioners to emphasize either aspect depending on the application. FREX thus surfaces words that are both common within a topic and rare elsewhere, providing a more discriminative and human‑friendly topic label than pure frequency lists.

The empirical evaluation consists of two parts. First, a large‑scale randomized experiment on Amazon Mechanical Turk compares human interpretability of topic summaries generated by FREX versus traditional frequency‑only lists. Participants consistently rated FREX‑based summaries as clearer and more distinctive, achieving a 23 % higher accuracy in identifying the correct topic and a 31 % reduction in decision time. Second, the authors benchmark the HPC model against standard LDA and hierarchical Dirichlet processes on synthetic and real‑world datasets. Results show that HPC yields substantially lower mean‑squared error in estimating exclusivity parameters (2–3× improvement) while maintaining comparable perplexity for overall word prediction. Moreover, the parallel HMC sampler processes a dataset of 2 million documents and 5 000 topics in under four hours, using roughly 40 % less memory than conventional Gibbs samplers.

The discussion highlights several strengths: (i) seamless integration of domain expertise via the taxonomy, (ii) a principled statistical treatment of both frequency and exclusivity, and (iii) scalability to industrial‑size corpora. Limitations include dependence on a well‑defined tree structure—if the taxonomy is misspecified or highly unbalanced, performance may degrade—and the need for careful tuning of the α parameter to match user preferences. Future work is suggested in extending the framework to non‑hierarchical multi‑label settings, learning the taxonomy jointly with the model, and incorporating multimodal data such as images or metadata.

In conclusion, the Hierarchical Poisson Convolution model together with the FREX scoring mechanism provides a robust, interpretable, and scalable solution for topic summarization. By moving beyond raw frequency and explicitly modeling word exclusivity within a hierarchical taxonomy, the approach delivers clearer topic descriptions, more accurate parameter estimates, and practical feasibility for large‑scale document analysis.