通过成功与失败案例对比，智能识别任务专家并优化训练监督。
TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation
- 基于成功/失败样本的专家激活差异，动态发现任务相关专家。
- 在错误样本中增强关键专家激活时的预测词监督，提升精度。
- 适用于多种MoE模型，特别适合需要精准专家调控的下游任务。
混合专家（MoE）语言模型将每个标记路由到少量专家，路由模式可用于下游适配中识别任务相关专家。然而现有方法存在两个局限：任务专家通常从聚合路由统计中识别，反映的是使用频率而非任务完成关联性；且任务专家激活信号未被充分用于监督分配。本文提出任务专家感知监督（TEXAS），结合正确性条件下的专家发现与标记级监督分配。TEXAS比较基模型成功与失败实例中的专家激活情况，保留在成功实例中更强激活的专家。微调时，在失败实例中当这些专家被激活时，会提升对应答案标记的监督权重。因此，TEXAS利用已有路由行为，无需限制适应到固定专家子集或强制目标路由分布。在三种MoE模型和六个基准上，TEXAS在18项设置中取得最佳或并列最佳表现，平均优于最强基线1.3–1.5分。消融实验与进一步分析验证了所发现专家及监督策略的有效性。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation. Yet current approaches have two limitations: task experts are typically identified from aggregate routing statistics that reflect usage rather than association with successful task completion, and task-expert activations remain underexplored as signals for supervision allocation. We introduce Task-Expert-Aware Supervision (TEXAS), which combines correctness-conditioned task expert discovery with token-level supervision allocation. TEXAS compares expert activations on instances that the base model solves successfully and those it fails to solve, and retains experts more strongly activated on successful instances. During fine-tuning, it upweights answer tokens in failed instances when they activate these experts. TEXAS therefore leverages existing routing behavior without restricting adaptation to a fixed expert subset or imposing an explicit target routing distribution. Across three MoE models and six benchmarks, TEXAS achieves the best or tied-best performance in 17 of 18 settings and improves over the strongest baseline by 1.3--1.5 points on average. Ablations and further analyses validate both the discovered experts and the resulting supervision strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。