On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks

Mixture-of-experts networks (MoEs) have demonstrated remarkable efficiency in modern deep learning. Despite their empirical success, the theoretical foundations underlying their ability to model complex tasks remain poorly understood. In this work, we conduct a systematic study of the expressive power of MoEs in modeling complex tasks with two common structural priors: low-dimensionality and sparsity. For shallow MoEs, we prove that they can efficiently approximate functions supported on low-dimensional manifolds, overcoming the curse of dimensionality. For deep MoEs, we show that $\mathcal{O}(L)$-layer MoEs with $E$ experts per layer can approximate piecewise functions comprising $E^L$ pieces with compositional sparsity, i.e., they can exhibit an exponential number of structured tasks. Our analysis reveals the roles of critical architectural components and hyperparameters in MoEs, including the gating mechanism, expert networks, the number of experts, and the number of layers, and offers natural suggestions for MoE variants.

💡 Research Summary

The paper provides a rigorous theoretical analysis of the expressive power of Mixture‑of‑Experts (MoE) networks when the target tasks exhibit two common structural priors: low‑dimensional manifolds and compositional sparsity.

Problem Setting and Background
MoE consists of a set of expert networks and a gating (routing) function that selects a small subset of experts for each input. While MoEs have become a core component of large language models and have shown impressive empirical performance, their theoretical capabilities have remained largely unexplored. The authors focus on two structural assumptions that are frequently observed in real data: (1) high‑dimensional data lie on a low‑dimensional smooth manifold, and (2) the target function can be expressed as a piecewise function where each piece depends only on a small subset of input coordinates (compositional sparsity).

Shallow MoE and Low‑Dimensional Manifolds
The authors first consider a compact d‑dimensional smooth manifold M embedded in ℝ^D. Using a finite atlas { (U_i, φ_i) }{i=1}^E and a partition of unity {ρ_i}, the target function f can be decomposed into local functions f|{U_i}∘φ_i^{-1} defined on the low‑dimensional domain