Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations

February 09, 2026

Reading time: 1 minute

...

📝 Original Info

Title: Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations
ArXiv ID: 2601.00457
Date: 2026-01-01
Authors: Hyunjun Kim

📝 Abstract

Mixture-of-Experts (MoE) models achieve efficiency through sparse activation, but the role of geometric regularization in expert specialization remains unclear. We apply orthogonality loss to enforce expert diversity and find it fails on multiple fronts: it does not reduce weight-space overlap (MSO actually increases by up to 114%), activation-space overlap remains high (∼0.6) regardless of regularization, and effects on performance are inconsistent-marginal improvement on WikiText-103 (-0.9%), slight degradation on TinyStories (+0.9%), and highly variable results on PTB (std > 1.0). Our analysis across 7 regularization strengths reveals no significant correlation (r = -0.293, p = 0.523) between weight and activation orthogonality. These findings demonstrate that weight-space regularization neither achieves its geometric goal nor reliably improves performance, making it unsuitable for MoE diversity.

📄 Full Content

...(본문 내용이 길어 생략되었습니다. 사이트에서 전문을 확인해 주세요.)

Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations

📝 Original Info

📝 Abstract

📄 Full Content

Table of Contents

Table of Contents

📝 Original Info

📝 Abstract

📄 Full Content

Start searching

No results found