动态调整专家权重，让多个模型协作实现无遗憾决策。
Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Making
- 用上下文驱动的专家投票机制，自动筛选最优专家组合。
- 理论证明算法在理想条件下可实现无遗憾学习，实测提升响应准确率。
- 适合需要实时优化大模型输出的场景，如智能客服、生成系统。
我们研究基于专家引导的在线多臂赌博机学习，称为在线专家混合（OMoE）。给定上下文，一组候选专家需决定如何聚合输出以最大化整体准确率。本文提出两种算法：第一种结合了聚合投票与基于UCB的成功淘汰机制，高效剔除表现较差的探索行为；第二种采用在线加权多数投票机制，使每位专家的投票权重与其预测能力成正比。我们在理想条件下推导出该设置下的后悔边界，并提供了相应的实验验证。作为现代应用实例，这些方法被用于一组专家大语言模型（LLMs）的在线微调，每次生成回应后，生成式LLM会动态重新分配专家权重和/或选择最优专家委员会，以生成最准确的回答。结果展示了新方法在整体性能上的提升，并提供了无遗憾的学习保证。
原文摘要 · Abstract (English)
We explore the use of expert-guided bandit learning, which we refer to as online mixture-of-experts (OMoE). In this setting, given a context, a candidate committee of experts must determine how to aggregate their outputs to achieve optimal results in terms of aggregate accuracy. We propose two algorithms to address this problem. The first algorithm combines aggregate voting with UCB-driven successive elimination, efficiently pruning suboptimal exploration actions. The second algorithm employs an online weighted-majority-voting mechanism, leveraging the respective voting power of each expert proportional to their predictive power. We derive theoretical guarantees for the regret properties in the bandit setting under ideal circumstances, and empirical results are provided accordingly. As a modern study on applications, these methods are applied to the online fine-tuning of a set of expert large language models (LLMs), where after each response, the generative LLM dynamically reweighs its set of experts and/or selects the optimal committee of experts to generate the most accurate response. Our results introduce new methodologies and no-regret guarantees for combining multiple experts to improve on the performance of the an aggregate model overall.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。