Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function

The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks to language modeling. Additionally, recent efforts have demonstrated that the sigmoid gate is provably more sample-efficient than its softmax counterpart under regression settings. Nevertheless, there are three notable concerns that have not been addressed in the literature, namely (i) the benefits of the sigmoid gate have not been established under classification settings; (ii) existing sigmoid-gated MoE models may not converge to their ground-truth; and (iii) the effects of a temperature parameter in the sigmoid gate remain theoretically underexplored. To tackle these open problems, we perform a comprehensive analysis of multinomial logistic MoE equipped with a modified sigmoid gate to ensure model convergence. Our results indicate that the sigmoid gate exhibits a lower sample complexity than the softmax gate for both parameter and expert estimation. Furthermore, we find that incorporating a temperature into the sigmoid gate leads to a sample complexity of exponential order due to an intrinsic interaction between the temperature and gating parameters. To overcome this issue, we propose replacing the vanilla inner product score in the gating function with a Euclidean score that effectively removes that interaction, thereby substantially improving the sample complexity to a polynomial order.

💡 Research Summary

The paper investigates the statistical properties of mixture‑of‑experts (MoE) models when the gating function is a sigmoid rather than the more common softmax, focusing on multinomial logistic (classification) tasks. Three gaps in the existing literature are addressed: (i) the lack of theoretical guarantees for sigmoid gating in classification, (ii) the failure of existing sigmoid‑gated MoE models to converge to the true mixing distribution when the number of experts is over‑specified, and (iii) the under‑explored impact of a temperature parameter on sample complexity.

Model and Modified Sigmoid Gate
Data consist of i.i.d. pairs ((X_i,Y_i)) with (X\in\mathbb{R}^d) and (Y\in