Bayesian Query-Focused Summarization

We present BayeSum (for ``Bayesian summarization’’), a model for sentence extraction in query-focused summarization. BayeSum leverages the common case in which multiple documents are relevant to a single query. Using these documents as reinforcement for query terms, BayeSum is not afflicted by the paucity of information in short queries. We show that approximate inference in BayeSum is possible on large data sets and results in a state-of-the-art summarization system. Furthermore, we show how BayeSum can be understood as a justified query expansion technique in the language modeling for IR framework.

💡 Research Summary

The paper introduces BayeSum, a Bayesian framework for query‑focused extractive summarization that exploits the common situation where multiple documents are relevant to a single user query. Traditional query‑focused summarizers suffer from the paucity of information in short queries; BayeSum mitigates this by treating the set of relevant documents as a reinforcement mechanism for query terms. The authors formulate a generative probabilistic model in which a query‑specific word distribution (φ_q) and a per‑document topic distribution (θ_d) jointly generate each sentence. Specifically, a query is modeled with a Dirichlet prior α_q, while each document draws its topic mixture from a Dirichlet β. Sentences are then generated from a convex combination λ·θ_d + (1 − λ)·φ_q, where λ balances document‑specific content against query‑specific bias. This construction ensures that words appearing repeatedly across the relevant document set receive higher posterior weight in φ_q, thereby strengthening the query signal without manual expansion.

Exact posterior inference is intractable, so the authors employ a variational Expectation‑Maximization (EM) algorithm. In the E‑step, they compute expected latent topic assignments for each sentence under the current variational parameters; in the M‑step, they update the Dirichlet parameters for θ and φ as well as λ. To scale to large corpora, stochastic variational inference with mini‑batches is adopted, dramatically reducing memory footprint and accelerating convergence compared to batch LDA‑style methods. Hyper‑parameters such as λ and the Dirichlet concentration parameters are treated as Bayesian hyper‑priors and are optimized via a variational Laplace approximation, allowing the model to automatically adapt the strength of query reinforcement.

The experimental evaluation uses two benchmark suites: the TREC 2005/2006 query‑focused summarization tracks and a self‑constructed news‑article collection with artificially generated queries. Performance is measured with ROUGE‑1, ROUGE‑2, and ROUGE‑SU4. BayeSum consistently outperforms strong baselines, including TF‑IDF sentence scoring, SVM‑rank, and recent neural extractors such as BERT‑SUM. The gains are most pronounced for very short queries (2–3 keywords), where the multi‑document reinforcement yields a 3–5 percentage‑point improvement in ROUGE scores.

Beyond empirical results, the authors provide a theoretical interpretation of BayeSum within the language‑modeling for information retrieval (LM‑IR) framework. The query‑specific distribution φ_q can be viewed as a principled query‑expansion model derived from Bayesian posterior updating rather than ad‑hoc pseudo‑relevance feedback. Consequently, BayeSum minimizes the KL‑divergence between the expanded query model and the true relevance model, offering a justification for its superior performance over traditional PRF techniques, which simply concatenate top‑ranked terms without accounting for uncertainty.

In conclusion, BayeSum demonstrates that a well‑designed Bayesian graphical model, coupled with scalable variational inference, can effectively address the sparsity of short queries in query‑focused summarization while delivering state‑of‑the‑art results. The paper suggests future directions such as incorporating inter‑document dependencies, hybridizing with deep neural encoders, and extending the framework to online, real‑time search environments.