arXiv:2510.12773cs.CLcs.AI2025-10被引 8

让大模型按需跳过或重复层,既省算力又提准确率。

Dr.LLM: Dynamic Layer Routing in LLMs

  • 给预训练模型加轻量路由,动态决定每层是否跳过、执行或重复。
  • 在逻辑和数学任务上提升准确率最高3.4个百分点,平均节省5层计算。
  • 无需重训,可迁移至多种新任务,适合追求高效推理的部署场景。

大型语言模型对每个输入标记都通过全部Transformer层,导致简单问题浪费算力,复杂问题缺乏深度推理灵活性。现有自适应深度方法依赖高成本推理搜索、架构改动或大规模重训练,常以牺牲准确率为代价。本文提出Dr. LLM,一种可回溯集成的框架,为预训练模型添加轻量级逐层路由模块,动态决定跳过、执行或重复某一层。路由模块通过蒙特卡洛树搜索(MCTS)获取高质量层配置,在算力预算下保持甚至提升准确率。设计包括窗口池化保障路由稳定、焦点损失缓解类别不平衡、瓶颈MLP路由增强鲁棒性。在ARC(逻辑)和DART(数学)任务上,准确率最高提升3.4个百分点,平均节省5层/例。在跨域任务(MMLU、GSM8k、AIME、TruthfulQA、SQuADv2、GPQA、PIQA、AGIEval)上仅损失0.85%准确率,仍保持效率优势,优于已有路由方法最高7.7个百分点。结果表明,显式监督的路由可不修改基础权重实现预算感知、准确率驱动的推理。

原文摘要 · Abstract (English)

Large Language Models (LLMs) process every token through all layers of a transformer stack, causing wasted computation on simple queries and insufficient flexibility for harder ones that need deeper reasoning. Adaptive-depth methods can improve efficiency, but prior approaches rely on costly inference-time search, architectural changes, or large-scale retraining, and in practice often degrade accuracy despite efficiency gains. We introduce Dr. LLM, Dynamic routing of Layers for LLMs, a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block. Routers are trained with explicit supervision: using Monte Carlo Tree Search (MCTS), we derive high-quality layer configurations that preserve or improve accuracy under a compute budget. Our design, windowed pooling for stable routing, focal loss with class balancing, and bottleneck MLP routers, ensures robustness under class imbalance and long sequences. On ARC (logic) and DART (math), Dr. LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average. Routers generalize to out-of-domain tasks (MMLU, GSM8k, AIME, TruthfulQA, SQuADv2, GPQA, PIQA, AGIEval) with only 0.85% accuracy drop while retaining efficiency, and outperform prior routing methods by up to +7.7%p. Overall, Dr. LLM shows that explicitly supervised routers retrofit frozen LLMs for budget-aware, accuracy-driven inference without altering base weights. Code is available at https://github.com/parameterlab/dr-llm.

大模型优化动态推理路由机制高效生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。