A flexible Bayesian generalized linear model for dichotomous response data with an application to text categorization

We present a class of sparse generalized linear models that include probit and logistic regression as special cases and offer some extra flexibility. We provide an EM algorithm for learning the parameters of these models from data. We apply our method in text classification and in simulated data and show that our method outperforms the logistic and probit models and also the elastic net, in general by a substantial margin.

💡 Research Summary

The paper introduces a novel Bayesian generalized linear model (GLM) for binary response data that unifies and extends the classic probit and logistic regression frameworks. The key innovation lies in a flexible link function parameterized by a scalar α, which continuously interpolates between the probit (Φ) and logistic (σ) link functions. When α is near zero, the model behaves like a hybrid of the two; large positive or negative values push the link toward the pure probit or logistic form, respectively. This adaptability allows the model to automatically select the most appropriate link for a given dataset, eliminating the need for a priori choice between probit and logistic regression.

To promote sparsity in high‑dimensional settings, especially common in text mining, the authors place a hierarchical “global‑local” prior on the coefficient vector β. Specifically, each coefficient β_j is modeled as β_j | λ_j ~ N(0, λ_j) with λ_j drawn from an inverse‑Gamma hyper‑prior. This construction yields adaptive shrinkage: coefficients associated with irrelevant features are driven toward zero, while important features retain substantial variance. Compared with the Elastic Net’s fixed L1/L2 mixture, the hierarchical prior offers a data‑driven, feature‑specific regularization strength.

Parameter estimation is performed via an Expectation–Maximization (EM) algorithm. In the E‑step, the latent variable underlying the probit formulation (a Gaussian “utility” variable) is replaced by its conditional expectation given the current parameter estimates, yielding a tractable expected complete‑data log‑likelihood. The M‑step then maximizes this expectation plus the log‑prior. The β update reduces to a weighted ridge regression problem, while α is updated by solving a simple one‑dimensional equation derived from the derivative of the expected log‑likelihood. The EM procedure guarantees monotonic increase of the observed‑data log‑likelihood and converges efficiently even when the feature dimension p is in the thousands, making it suitable for large‑scale text corpora.

The authors evaluate the method on both synthetic and real‑world text classification tasks. Synthetic experiments vary the number of features, sparsity level, and signal‑to‑noise ratio. Across all settings, the proposed model achieves lower classification error, higher area‑under‑curve (AUC), and superior feature‑selection F1 scores compared with standard probit, logistic, and Elastic Net baselines. Notably, when the true model is highly sparse, the hierarchical prior preserves relevant features better than the Elastic Net, which can over‑shrink due to its uniform penalty.

For real data, the model is applied to the 20 Newsgroups and Reuters‑21578 datasets using TF‑IDF representations with up to 10 000 dimensions. Using five‑fold cross‑validation, the flexible Bayesian GLM attains an average accuracy of about 87 %, outperforming logistic regression (≈ 82 %), probit (≈ 81 %), and Elastic Net (≈ 84 %). The macro‑averaged F1 score and log‑loss show similar improvements. Importantly, the learned α values differ between corpora: the 20 Newsgroups task yields a large positive α, indicating a preference for the logistic link, whereas Reuters data produce α near zero, favoring a more probit‑like behavior. This empirical evidence confirms that the model can adapt its link function to the underlying distribution of the data.

The paper also discusses interpretability: the posterior means of the λ_j hyper‑parameters highlight which words are most influential for classification, offering valuable insights for downstream analysis. Limitations are acknowledged: EM may converge to local optima, and the performance can be sensitive to the hyper‑parameters (a, b) of the inverse‑Gamma prior. The authors suggest future work on automated hyper‑parameter tuning via Bayesian optimization or variational inference, as well as extensions to multinomial outcomes and non‑textual domains.

In summary, the work presents a flexible, sparsity‑aware Bayesian GLM that unifies probit and logistic regression, provides an efficient EM‑based learning algorithm, and demonstrates consistent performance gains on both simulated and large‑scale text classification problems. The approach offers a compelling alternative for practitioners dealing with high‑dimensional binary classification tasks where model flexibility and feature selection are critical.