arXiv:2503.05641cs.CLcs.AI2025-03被引 28

根据问题自动匹配所需技能,精准调用专家模型提升推理能力。

Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills

  • 通过分析问题推断出数学、逻辑等技能,动态选择最相关专家
  • 在多个基准上平均性能提升8.15%,且仅用单卡实现
  • 无需多轮对话,适合快速推理与未见任务泛化

结合预训练大模型是应对多样化推理任务的有前景方法。然而,传统的任务级专家选择过于粗粒度,不同实例可能需要不同专业能力。为此,我们提出Skill-MoE:一种基于符号化技能、无梯度的混合专家框架,实现实例级专家选择。该框架从每个查询中推断技能(如数学中的代数),依据技能相关性选择专家,并由各专家独立生成推理过程。最终的k个输出由擅长整合多样回答的聚合器合成。尽管实例级选择显著提升性能,但直接实现会因频繁加载模型带来巨大开销。我们采用批处理策略,按分配的专家分组实例,使每模型仅加载一次。结果,Skill-MoE可在单张GPU上集成16个专家模型,运行时间与使用4张GPU的前序多代理基线相当。在MMLU-Pro、GPQA、AIME和MedMCQA等多个基准上,其平均绝对性能提升达8.15%。该方法还能良好泛化至未见任务,优于依赖多轮交互的讨论式方法,且无需昂贵的交互成本。

原文摘要 · Abstract (English)

Combining existing pre-trained LLMs is a promising approach for diverse reasoning tasks. However, task-level expert selection is often too coarse-grained, since different instances may require different expertise. To address this, we propose Skill-MoE, a symbolic, skill-based, and gradient-free Mixture-of-Experts framework for instance-level expert selection. Skill-MoE infers skills (e.g., algebra in mathematics) from each query, selects experts based on skill relevance, and lets each expert generate its own reasoning. The resulting k outputs are then synthesized by an aggregator chosen for its ability to integrate diverse responses. While instance-level selection substantially improves performance, naively implementing it incurs heavy overhead from repeated model loading and offloading. We address this with a batch inference strategy that groups instances by assigned experts, allowing each model to be loaded only once. As a result, Skill-MoE integrates 16 expert models on a single GPU with runtime comparable to prior multi-agent baselines using 4 GPUs. Across diverse benchmarks (MMLU-Pro, GPQA, AIME, and MedMCQA), Skill-MoE achieves an average absolute improvement of 8.15% over the best baseline. It also generalizes well to unseen tasks and outperforms discussion-based methods without requiring expensive multi-round interactions.

混合专家推理增强技能识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。