用拍卖机制解决专家模型遗忘问题,实现稳定持续学习。
MoB: Mixture of Bidders
- 用维克里拍卖代替传统门控网络,让专家自主竞价
- 在分拆MNIST上达88.77%准确率,是基线的4.5倍
- 无需预设任务边界,专家可自动识别知识边界
Mixture of Experts(MoE)架构在扩展神经网络方面表现卓越,但在持续学习中因门控网络易受灾难性遗忘而受限。本文提出新型框架Mixture of Bidders(MoB),将专家路由重构为去中心化的经济机制。MoB以维克里-克拉克-格罗夫斯(VCG)拍卖取代学习型门控网络,专家通过出价自身真实成本(预测损失与弹性权重巩固惩罚之和)竞争数据批次。该博弈论方法具备三大优势:(1) 无状态路由,免疫灾难性遗忘;(2) 真实出价由主导策略激励相容保证;(3) 无需显式任务边界即可实现自发专业化。在Split-MNIST基准上,MoB平均准确率达88.77%,相较最强基线Gated MoE的19.54%提升4.5倍。此外,我们引入自监控专家,可自主识别知识固化边界,无需人工划分任务。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) architectures have demonstrated remarkable success in scaling neural networks, yet their application to continual learning remains fundamentally limited by a critical vulnerability: the learned gating network itself suffers from catastrophic forgetting. We introduce Mixture of Bidders (MoB), a novel framework that reconceptualizes expert routing as a decentralized economic mechanism. MoB replaces learned gating networks with Vickrey-Clarke-Groves (VCG) auctions, where experts compete for each data batch by bidding their true cost -- a principled combination of execution cost (predicted loss) and forgetting cost (Elastic Weight Consolidation penalty). This game-theoretic approach provides three key advantages: (1) {stateless routing that is immune to catastrophic forgetting, (2) \textbf{truthful bidding} guaranteed by dominant-strategy incentive compatibility, and (3) emergent specialization without explicit task boundaries. On Split-MNIST benchmarks, MoB achieves 88.77% average accuracy compared to 19.54% for Gated MoE and 27.96% for Monolithic EWC, representing a 4.5 times improvement over the strongest baseline. We further extend MoB with autonomous self-monitoring experts that detect their own knowledge consolidation boundaries, eliminating the need for explicit task demarcation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。