让大模型根据输入自动跳过冗余计算层，提升效率且不丢质量。
LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models
- 按输入类型动态决定跳过哪些推理层，用轻量适配器实现
- 工具调用阶段可省15.25%算力，推理阶段省2.34%，总参数仅110万
- 适合追求高效推理的智能体系统，特别在资源受限场景下
智能体语言模型系统包含两类结构迥异的步骤：结构化工具调用（短、确定、低困惑度）和开放式规划/推理（长、复杂、高困惑度）。当前推理系统对所有步骤采用相同算力。本文提出LayerRoute，一种轻量级适配器，可基于输入选择性跳过Transformer层。该方法在Qwen2.5-0.5B-Instruct的24个Transformer层中每层添加：(1) 每层路由器（约897参数，Linear(896,1)），通过直通估计器输出硬二值门控；(2) 在Q/K/V/O注意力投影上使用LoRA适配器（秩为8，约1.08M参数）。主干权重保持冻结。在代理数据集（Hermes、Glaive、GSM8K、Turing）上进行一次端到端训练（3,000步，A100 40GB上耗时6.4分钟），加入门控正则项，促使系统学习不同输入类型的可跳过层。训练后，工具调用阶段跳过15.25%的浮点运算，规划阶段仅跳过2.34%，总可训练参数仅110万（占494M主干的0.22%）。由于LoRA微调，模型质量反而提升：工具调用困惑度降低-1.29，规划阶段降低-1.30。
原文摘要 · Abstract (English)
Agentic language model systems alternate between two structurally distinct step types: structured tool calls (short, deterministic, low perplexity) and open-ended planning/reasoning steps (long, complex, high perplexity). Despite this heterogeneity, current inference systems apply identical compute to every step. We introduce LayerRoute, a lightweight adapter that learns to selectively skip transformer blocks on a per-input basis. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with: (1) a per-layer router (~897 parameters, Linear(896,1)) that outputs a hard binary gate via the straight-through estimator, and (2) LoRA adapters (rank 8, ~1.08M parameters) on the Q/K/V/O attention projections. The backbone weights remain frozen. A single end-to-end training pass on agentic data (Hermes, Glaive, GSM8K, Turing) with a gate regularisation term forces the system to discover which blocks are skippable per input type. After 3,000 steps (6.4 minutes on an A100 40GB), LayerRoute achieves a 12.91% skip differential: tool calls skip 15.25% of FLOPs while planning steps skip only 2.34%, using only 1.10M trainable parameters (0.22% of the 494M backbone). Quality improves over the base model due to LoRA adaptation, with perplexity delta of -1.29 on tool calls and -1.30 on planning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。