MoB: Mixture of Bidders

Mixture of Experts (MoE) architectures have demonstrated remarkable success in scaling neural networks, yet their application to continual learning remains fundamentally limited by a critical vulnerability: the learned gating network itself suffers from catastrophic forgetting. We introduce Mixture of Bidders (MoB), a novel framework that reconceptualizes expert routing as a decentralized economic mechanism. MoB replaces learned gating networks with Vickrey-Clarke-Groves (VCG) auctions, where experts compete for each data batch by bidding their true cost – a principled combination of execution cost (predicted loss) and forgetting cost (Elastic Weight Consolidation penalty). This game-theoretic approach provides three key advantages: (1) {stateless routing that is immune to catastrophic forgetting, (2) \textbf{truthful bidding} guaranteed by dominant-strategy incentive compatibility, and (3) emergent specialization without explicit task boundaries. On Split-MNIST benchmarks, MoB achieves 88.77% average accuracy compared to 19.54% for Gated MoE and 27.96% for Monolithic EWC, representing a 4.5 times improvement over the strongest baseline. We further extend MoB with autonomous self-monitoring experts that detect their own knowledge consolidation boundaries, eliminating the need for explicit task demarcation.

💡 Research Summary

The paper tackles a critical bottleneck in continual learning for Mixture‑of‑Experts (MoE) architectures: the learned gating network itself suffers catastrophic forgetting, which destroys the routing logic that enables expert specialization. The authors term this the “gater forgetting problem” and demonstrate, both conceptually and experimentally, that even when individual experts retain their knowledge, a corrupted gate prevents the system from accessing the right expert for earlier tasks.

To eliminate this vulnerability, the authors propose Mixture of Bidders (MoB), a framework that replaces the learned gate with a stateless Vickrey‑Clarke‑Groves (VCG) auction mechanism. For each incoming data batch, every expert computes a bid equal to a weighted sum of two components: (1) an execution cost, defined as the predicted loss on the batch, and (2) a forgetting cost, estimated via Elastic Weight Consolidation (EWC) using the Fisher information matrix. Hyperparameters α and β control the relative importance of these components. The VCG auction selects the expert with the lowest bid (the “winner”) and charges it the second‑lowest bid, guaranteeing dominant‑strategy incentive compatibility (DSIC). This property forces truthful revelation of each expert’s true processing cost, ensuring that the most competent expert for a given batch is always chosen.

Because the VCG auction is a fixed rule without learnable parameters, routing is immutable throughout training—there is no “gate” that can forget. The winner trains on the batch with standard EWC regularization, and after a task (or a self‑detected consolidation point) the expert updates its Fisher matrix. The authors further extend MoB with a self‑monitoring mechanism: each expert tracks the variance‑to‑mean ratio of its execution cost over a sliding window. When this coefficient of variation falls below a threshold, the expert autonomously triggers Fisher updates, effectively detecting task boundaries without external signals. An EMA‑based spike detector also flags sudden distribution shifts to prevent “muddy” Fisher matrices.

Empirical evaluation on the Split‑MNIST benchmark (five sequential binary digit tasks) shows dramatic gains. MoB achieves an average accuracy of 88.77 % (±0.0459), compared to 19.54 % for a Gated MoE, 27.96 % for a monolithic EWC model, 46.24 % for random expert assignment, and 19.80 % for naive fine‑tuning. The improvement is statistically significant (p < 0.001). Per‑task analysis reveals that early tasks retain >99 % accuracy after training on later tasks, whereas baselines suffer severe recency bias. Expert win‑rate statistics illustrate emergent specialization: without any explicit task labels or auxiliary loss, experts naturally gravitate toward subsets of tasks (e.g., one expert handling 40 % of batches from later tasks). This confirms that truthful bidding drives effective allocation.

A further experiment on the CORe50 continual vision stream validates the self‑monitoring extension: the system autonomously detects distribution changes and updates Fisher matrices, maintaining stable performance without predefined task boundaries.

The discussion emphasizes three pillars of MoB’s success: (1) stateless routing eliminates forgetting at the routing layer; (2) DSIC guarantees that experts reveal true costs, turning the routing problem into a principled economic allocation; (3) dynamic forgetting awareness via the bidding function integrates knowledge protection directly into the decision process. The authors also note that MoB’s decisions are interpretable—each routing choice can be explained by the constituent execution and forgetting costs—unlike opaque learned gates. Scalability is addressed conceptually: the modular expert pool can host heterogeneous architectures (e.g., transformers), suggesting feasibility for trillion‑parameter language models. Future work includes scaling MoB to continual pre‑training of large language models and exploring richer auction designs (e.g., combinatorial auctions) for multi‑expert batch assignments.

In summary, MoB reframes expert selection in MoE as a truthful, stateless auction, thereby solving the gater forgetting problem that has limited MoE’s applicability to lifelong learning. The approach delivers strong empirical performance, theoretical guarantees, and interpretability, opening a promising avenue for robust, scalable continual learning systems.