Bayesian multinomial regression with class-specific predictor selection

Consider a multinomial regression model where the response, which indicates a unit’s membership in one of several possible unordered classes, is associated with a set of predictor variables. Such models typically involve a matrix of regression coefficients, with the $(j,k)$ element of this matrix modulating the effect of the $k$th predictor on the propensity of the unit to belong to the $j$th class. Thus, a supposition that only a subset of the available predictors are associated with the response corresponds to some of the columns of the coefficient matrix being zero. Under the Bayesian paradigm, the subset of predictors which are associated with the response can be treated as an unknown parameter, leading to typical Bayesian model selection and model averaging procedures. As an alternative, we investigate model selection and averaging, whereby a subset of individual elements of the coefficient matrix are zero. That is, the subset of predictors associated with the propensity to belong to a class varies with the class. We refer to this as class-specific predictor selection. We argue that such a scheme can be attractive on both conceptual and computational grounds.

💡 Research Summary

The paper addresses the problem of variable selection in multinomial (unordered‑category) regression from a Bayesian perspective. In the conventional formulation, the response belongs to one of J classes and the relationship with K predictors is captured by a J × K coefficient matrix β, where each column corresponds to a predictor that influences all classes equally. Bayesian variable selection therefore proceeds by placing a spike‑and‑slab prior on entire columns of β, effectively turning predictors on or off globally (global predictor selection, GPS).

The authors argue that this global approach is often unrealistic because a predictor may be relevant for only a subset of the classes. To accommodate such heterogeneity they propose class‑specific predictor selection (CSPS). For each element β_{jk} they introduce a binary inclusion indicator γ_{jk}. When γ_{jk}=0 the coefficient is forced to zero (the “spike”), and when γ_{jk}=1 the coefficient follows a diffuse normal prior (the “slab”). The inclusion probabilities π_k govern the overall propensity of predictor k to be selected across classes and are given a Beta hyper‑prior. A hierarchical prior also includes a variance hyper‑parameter σ² with an inverse‑Gamma prior.

Posterior inference is carried out by a Gibbs sampler (or Metropolis‑in‑Gibbs) that alternately updates the binary indicators and the non‑zero coefficients. Conditional on γ, the β_{jk} have conjugate normal posteriors; conditional on β, the γ_{jk} have Bernoulli posteriors with probabilities that depend on the likelihood contribution of each class‑predictor pair. This block‑wise updating yields efficient mixing even when the total number of potential coefficients (J × K) is large, because most γ_{jk} converge to zero, rendering the effective model sparse.

The authors evaluate CSPS through extensive simulation studies. They compare CSPS with the traditional GPS under two regimes: (i) class‑specific effects (different predictors matter for different classes) and (ii) homogeneous effects (the same predictors matter for all classes). In the heterogeneous scenario CSPS dramatically outperforms GPS in terms of true‑positive rate, false‑positive rate, and overall classification accuracy, while in the homogeneous case the two methods perform comparably. The simulations also demonstrate that CSPS reduces over‑fitting by eliminating irrelevant coefficients, leading to more stable posterior predictive distributions.

Two real‑world applications illustrate the practical benefits. In a medical diagnostic dataset, CSPS identifies biomarkers that are predictive only for particular disease subtypes, whereas GPS either discards them entirely or forces them into all subtypes, obscuring the nuanced biology. In a text‑classification task, CSPS pinpoints words that are discriminative for specific topics, providing clearer interpretability for end‑users. In both cases, model‑averaged predictions derived from the posterior distribution of γ and β show lower misclassification rates than competing multinomial logistic models that use either LASSO regularization or global Bayesian variable selection.

From a computational standpoint, the CSPS algorithm scales linearly with the number of class‑predictor pairs, and the sparsity induced by the spike‑and‑slab prior leads to substantial memory savings. Moreover, the block Gibbs updates are amenable to parallel implementation, making the approach feasible for moderate‑to‑large J and K settings.

The discussion acknowledges limitations: the current formulation assumes a linear log‑odds link and does not directly accommodate interactions or non‑linear transformations. Extending the framework to hierarchical or deep models would require more sophisticated priors and sampling schemes. The authors also suggest future work on adaptive hyper‑priors for π_k, incorporation of covariate‑dependent inclusion probabilities, and exploration of variational approximations for even larger datasets.

In summary, the paper introduces a novel Bayesian multinomial regression framework that allows predictor inclusion to vary across outcome classes. By placing a spike‑and‑slab prior on individual matrix elements rather than on whole columns, the method achieves finer‑grained variable selection, improves predictive performance when class‑specific effects exist, and retains the full benefits of Bayesian model averaging. The methodological contributions are supported by rigorous simulation evidence and compelling real‑data examples, positioning CSPS as a valuable addition to the toolbox of statisticians and data scientists working with categorical outcomes.