arXiv:2607.26052cs.LG2026-07

根据模型不确定度动态分配专家,让难样本获得更多计算资源。

Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA

论文配图:Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA
图 1 · 摘自论文原文
  • 按路由置信度动态激活专家,难样本调用更多专家
  • 在相同算力下提升8个常识任务表现,且激活专家更少
  • 无需额外参数,适合需要高效推理的模型微调场景

低秩适配(LoRA)的专家混合(MoE)模型通常将每个标记固定分配给k个专家,但不同标记的模型不确定性不同,固定k值会浪费算力于简单标记,忽视困难标记。本文发现路由输出分布本身可作为每标记的不确定性信号:集中分布表示高置信,平坦分布表示模糊。提出CARE(置信度自适应专家路由),以核心方式激活专家:按路由权重降序激活,直至累积质量达阈值,并在专家意见分歧时稍作扩展。预算恒温器调节阈值,使平均活跃专家数匹配目标。CARE为无参、单前向传播的即插即用规则。在LLaMA-3.1-8B和Qwen2.5-7B上八个常识基准测试中,其性能优于固定top-k MoE-LoRA,在相同算力下达成与k=4基线相当效果,同时激活更少专家。相同信号也提升分布外检测能力,优于MSP、熵和多轮代理。通过核心保真度、预算最优性和分歧的主观认知解释支持设计,并开源代码。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA) route every token to a fixed number of experts $k$. Tokens differ in how uncertain the model is about them, so a single k over-spends on easy tokens and under-serves hard ones. We observe that the router's output distribution is already a per-token uncertainty signal: peaked mass indicates confidence, while a flat distribution indicates ambiguity. We introduce CARE (Confidence-Adaptive Routing of Experts), which admits experts in a nucleus fashion. Experts are activated in decreasing router weight until their cumulative mass reaches a threshold, with a small extension when the admitted experts disagree. A budget thermostat calibrates the threshold so that the average number of active experts matches any target. CARE is a drop-in, single-forward-pass rule with no extra parameters. Across eight commonsense benchmarks on LLaMA-3.1-8B and Qwen2.5-7B, as well as math, code, and knowledge tasks, CARE improves over fixed top-k MoE-LoRA at matched compute and matches the fixed-k=4 baseline while activating fewer experts. The same confidence and disagreement signals also improve out-of-distribution detection over MSP, entropy, and multi-pass proxies. We support the design with nucleus fidelity, budget optimality, and an epistemic reading of disagreement, and we release code.

MoELoRA专家路由自适应计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。