arXiv:2608.09444cs.LGcs.CL2026-08

让循环语言模型按需计算,显著提升推理效率。

Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching

论文配图:Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
图 1 · 摘自论文原文
  • 按迭代步粒度调度,实现深度自适应推理
  • 实测吞吐提升1.5-1.9倍,延迟降低45%-90%
  • 适合高负载动态服务场景下的高效推理

循环语言模型的核心优势在于深度自适应推理:通过可变次数地重复共享层,对简单词元使用较少计算,对复杂词元使用更多计算。然而,这种自适应性破坏了传统批处理机制——同一批次内的词元需要不同次数的迭代,无法统一前向传播,导致高效推理困难。标准推理框架如vLLM以词元为单位调度,无法在前向过程中移除词元。虽有环级调度方案被提出,但从未实现端到端。其核心挑战在于循环结构包含非循环的边界阶段(如词元嵌入和语言模型头部),需与循环步骤采用不同调度频率。本文提出连续深度批处理(CDB),以单次迭代为粒度进行调度。CDB将边界阶段与循环步骤分别置于独立优先队列中,提前一步做出退出决策,并将所有调度工作与GPU计算重叠。在Ouro 1.4B和Huginn 3.5B模型上,CDB可实现高达99%的理论最大加速比,使离线吞吐提升1.5-1.9倍,动态服务负载下归一化延迟降低45%-90%。

原文摘要 · Abstract (English)

A main promise of looped language models (LMs) is depth-adaptive inference. By iterating a block of shared layers a variable number of times, the model can use less compute for "easy" tokens and more for "hard" ones. However, this adaptivity breaks standard batching: tokens in the same batch now require a different number of loops, so there is no unified forward pass, making efficient inference difficult. Standard inference frameworks like vLLM schedule on the token level and cannot handle this because tokens need to be removed from the batch within the forward pass. Loop-level scheduling has been proposed as a solution, but never implemented end to end. The key challenge is that looped architectures also contain non-looped boundary stages (e.g., token embedding and LM head) that must be scheduled at different frequencies than the loop. We introduce continuous depth batching (CDB), which schedules at the granularity of individual loop iterations. CDB handles boundary stages and loop steps in separate priority queues, makes exit decisions one step ahead, and overlaps all scheduling work with GPU computation. On Ouro 1.4B and Huginn 3.5B, CDB can realize up to $99\%$ of the theoretical maximum speed-up from adaptive-depth, translating to $1.5$-$1.9\times$ higher offline throughput and $45$-$90\%$ lower normalized latency under dynamic serving load.

自适应推理循环模型批量调度高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。