让大模型按需分配计算深度,更省力更聪明。
Dynamic Reasoning Chains through Depth-Specialized Mixture-of-Experts in Transformer Architectures
- 根据输入复杂度动态选择不同深度的专家模块,实现智能推理链组装。
- 相比固定深度模型,计算节省16%,推理快35%,复杂任务准确率高2.8%。
- 推理过程可解释,适合追求效率与透明性的大模型应用。
当前Transformer架构对所有输入采用相同的处理深度,导致资源浪费并限制推理质量。简单事实查询与复杂逻辑问题均经历多层计算,既低效又制约深度推理。为此,我们提出深度专用混合专家(DS-MoE)框架,将MoE范式从宽度扩展到深度专用计算。该框架引入专为不同推理深度优化的专家模块:浅层模式识别、组合推理、逻辑推断、记忆整合和元认知监督。一个学习到的路由网络动态构建定制推理链,仅激活匹配输入复杂度所需的专家。模型在800GB的The Pile数据集上训练与评估,涵盖科学论文、法律文本、编程代码和网页内容等多元领域,支持跨推理深度的系统性评估。实验表明,DS-MoE相较均匀深度模型实现最高16%的计算节省和35%的推理加速,同时在复杂多步推理基准上准确率提升2.8%。此外,路由决策生成可解释的推理链,增强透明性与可扩展性。这些结果证明,深度专用模块化处理能同步提升大规模语言模型的效率、推理质量和可解释性。
原文摘要 · Abstract (English)
Contemporary transformer architectures apply identical processing depth to all inputs, creating inefficiencies and limiting reasoning quality. Simple factual queries are subjected to the same multilayered computation as complex logical problems, wasting resources while constraining deep inference. To overcome this, we came up with a concept of Dynamic Reasoning Chains through Depth Specialised Mixture of Experts (DS-MoE), a modular framework that extends the Mixture of Experts paradigm from width-based to depth specialised computation. DS-MoE introduces expert modules optimised for distinct reasoning depths, shallow pattern recognition, compositional reasoning, logical inference, memory integration, and meta-cognitive supervision. A learned routing network dynamically assembles custom reasoning chains, activating only the necessary experts to match input complexity. The dataset on which we trained and evaluated DS-MoE is on The Pile, an 800GB corpus covering diverse domains such as scientific papers, legal texts, programming code, and web content, enabling systematic assessment across reasoning depths. Experimental results demonstrate that DS-MoE achieves up to 16 per cent computational savings and 35 per cent faster inference compared to uniform-depth transformers, while delivering 2.8 per cent higher accuracy on complex multi-step reasoning benchmarks. Furthermore, routing decisions yield interpretable reasoning chains, enhancing transparency and scalability. These findings establish DS-MoE as a significant advancement in adaptive neural architectures, demonstrating that depth-specialised modular processing can simultaneously improve efficiency, reasoning quality, and interpretability in large-scale language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。