"Look Ma, No Hands!" A Parameter-Free Topic Model

It has always been a burden to the users of statistical topic models to predetermine the right number of topics, which is a key parameter of most topic models. Conventionally, automatic selection of this parameter is done through either statistical model selection (e.g., cross-validation, AIC, or BIC) or Bayesian nonparametric models (e.g., hierarchical Dirichlet process). These methods either rely on repeated runs of the inference algorithm to search through a large range of parameter values which does not suit the mining of big data, or replace this parameter with alternative parameters that are less intuitive and still hard to be determined. In this paper, we explore to “eliminate” this parameter from a new perspective. We first present a nonparametric treatment of the PLSA model named nonparametric probabilistic latent semantic analysis (nPLSA). The inference procedure of nPLSA allows for the exploration and comparison of different numbers of topics within a single execution, yet remains as simple as that of PLSA. This is achieved by substituting the parameter of the number of topics with an alternative parameter that is the minimal goodness of fit of a document. We show that the new parameter can be further eliminated by two parameter-free treatments: either by monitoring the diversity among the discovered topics or by a weak supervision from users in the form of an exemplar topic. The parameter-free topic model finds the appropriate number of topics when the diversity among the discovered topics is maximized, or when the granularity of the discovered topics matches the exemplar topic. Experiments on both synthetic and real data prove that the parameter-free topic model extracts topics with a comparable quality comparing to classical topic models with “manual transmission”. The quality of the topics outperforms those extracted through classical Bayesian nonparametric models.

💡 Research Summary

The paper tackles a long‑standing usability problem in topic modeling: the need for users to pre‑specify the number of topics (K). Traditional approaches either run the model repeatedly with different K values and select the best using held‑out likelihood, perplexity, AIC, BIC, etc., or adopt Bayesian non‑parametric priors such as the hierarchical Dirichlet process (HDP) that replace K with a concentration parameter. Both strategies either incur heavy computational cost or introduce new hyper‑parameters that are equally hard to set, and the Bayesian non‑parametric models often produce lower‑quality topics compared with classic parametric models like PLSA or LDA.

The authors propose a non‑parametric extension of Probabilistic Latent Semantic Analysis (PLSA), called nPLSA, which dynamically grows the topic set during a single EM run. The key idea is to measure how well a document d is explained by the current topic set Θ using a log‑likelihood ratio:

Δ(d, Θ) = log p(w_d | θ_d) − log p(w_d | Θ),

where θ_d is the empirical language model of document d. If Δ exceeds a threshold ε, the document “promotes” itself to a new topic; otherwise the existing topics are used to infer the posterior over latent variables. This decision is embedded in the E‑step, and the M‑step updates the topic–word and document–topic distributions as in ordinary PLSA. Consequently, nPLSA can explore a whole range of K values without restarting the algorithm.

Because ε itself is a new hyper‑parameter, the paper introduces two truly parameter‑free strategies to eliminate it:

1. Diversity‑based stopping – After each iteration the algorithm computes a diversity score among the current topics (e.g., average pairwise cosine distance or KL divergence). As K increases, diversity initially rises (new topics are distinct) but eventually falls when topics become redundant. The algorithm stops when diversity reaches its maximum, which empirically corresponds to the optimal K for the data.

2. Weak supervision via an exemplar topic – Users provide a single keyword or short phrase representing the granularity they desire. The system adjusts ε so that the resulting topics have a similarity to the exemplar that matches the user’s intent. This approach lets users control granularity without having to guess a numeric K.

Both strategies retain the simplicity of the original EM loop; the only extra computation is the diversity measurement or a lightweight adjustment of ε based on the exemplar. No sophisticated variational inference or Gibbs sampling is required, making the method scalable to large corpora.

The authors evaluate nPLSA on synthetic data (where the true K is known) and on real‑world collections such as news articles, Wikipedia pages, and social‑media posts. They report three main metrics:

• Perplexity – nPLSA’s perplexity is comparable to or slightly better than standard PLSA with a manually chosen K.
• Topic coherence – Measured by normalized pointwise mutual information (NPMI) or similar scores, nPLSA consistently outperforms HDP and other Bayesian non‑parametric baselines.
• Human judgment – Expert annotators rate the interpretability of topics; nPLSA’s topics are judged more coherent and meaningful.

The diversity‑based stopping criterion accurately recovers the true number of topics in synthetic experiments and selects a sensible K in real data, aligning with the point where the coherence curve plateaus. The exemplar‑based method allows users to retrieve topics at the desired granularity (e.g., “machine learning” yields related topics like “information retrieval” and “data mining” but not overly fine‑grained sub‑topics).

In terms of efficiency, nPLSA runs a single EM pass, whereas traditional model‑selection pipelines require dozens of runs, and Bayesian non‑parametric models need costly Gibbs sampling or variational updates. Empirically, nPLSA is 5–10× faster than these alternatives while delivering higher‑quality topics.

The paper concludes that a non‑parametric PLSA with either diversity‑driven or exemplar‑driven stopping provides a truly parameter‑free, easy‑to‑implement, and scalable solution for topic modeling. Future work suggested includes integrating word embeddings, extending to online streaming settings, and applying the framework to hierarchical topic discovery.