通过动态调控专家数量提升大模型推理能力,无需额外计算开销。
Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time
- 测试时动态调整激活专家数,生成多样化推理路径。
- 在数学、代码、知识等任务上显著提升准确率与稳定性。
- 适合追求高效推理优化的模型部署与研究者使用。
测试时扩展(TTS)通过推理阶段增加计算量来增强大语言模型(LLM)的推理能力。然而,现有方法主要依赖输出层面的采样,忽略了模型架构的作用。在主流混合专家(MoE)LLM中,我们发现改变激活专家数量会生成互补的解集且准确率稳定,揭示了一种新的、未被充分探索的多样性来源。受此启发,我们提出动态专家搜索(DES),一种将专家激活纳入可控制搜索空间的TTS策略。DES包含两个核心组件:(1) 动态MoE,可在推理时直接控制专家数量,生成多样化推理轨迹而无需额外成本;(2) 专家配置继承,保持单条推理路径内专家数量一致,跨运行间变化,从而平衡稳定性与多样性。在多种MoE架构、验证器及推理基准(如数学、代码和知识)上的广泛实验表明,DES稳定优于基线方法,在不增加成本的情况下提升准确率与稳定性。结果表明,DES是一种实用且可扩展的架构感知型TTS,展示了现代LLM结构灵活性如何推动推理能力进步。
原文摘要 · Abstract (English)
Test-Time Scaling (TTS) enhances the reasoning ability of large language models (LLMs) by allocating additional computation during inference. However, existing approaches primarily rely on output-level sampling while overlooking the role of model architecture. In mainstream Mixture-of-Experts (MoE) LLMs, we observe that varying the number of activated experts yields complementary solution sets with stable accuracy, revealing a new and underexplored source of diversity. Motivated by this observation, we propose Dynamic Experts Search (DES), a TTS strategy that elevates expert activation into a controllable dimension of the search space. DES integrates two key components: (1) Dynamic MoE, which enables direct control of expert counts during inference to generate diverse reasoning trajectories without additional cost; and (2) Expert Configuration Inheritance, which preserves consistent expert counts within a reasoning path while varying them across runs, thereby balancing stability and diversity throughout the search. Extensive experiments across MoE architectures, verifiers and reasoning benchmarks (i.e., math, code and knowledge) demonstrate that DES reliably outperforms TTS baselines, enhancing accuracy and stability without additional cost. These results highlight DES as a practical and scalable form of architecture-aware TTS, illustrating how structural flexibility in modern LLMs can advance reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。