arXiv:2606.09514cs.LG2026-06

动态调整模型深度,按预算精准控制推理成本

BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Model Inference

论文配图:BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Model Inference
图 1 · 摘自论文原文
  • 根据输入动态选择层数,用轻量模块决定跳过哪些层
  • 在相同预算下比固定剪枝更准,且支持解码时重新路由
  • 适合需要严格控算力或灵活适配不同场景的部署应用

大型语言模型因深度和参数量大导致推理成本高。深度剪枝可通过跳过冗余Transformer层降低延迟,但现有方法(一)在用户指定计算预算时控制能力有限,(二)通常固定路由路径,无法随解码过程中上下文增长而自适应。我们提出Buddy,一种基于预算驱动的动态深度路由框架。Buddy使用轻量决策模块,根据输入对中间层评分,并确定性地执行前k层以满足给定预算。为支持解码时自适应,Buddy复用第一层的键值缓存作为低开销全局上下文源,并在每次路由决策前将其与最新词元表示融合。当无显式预算时,可选的预算预测器会估计依赖输入的计算层级,以平衡质量与效率。在Llama系列和Qwen模型上的实验表明,Buddy在性能上优于强静态剪枝基线,常能提升准确率-计算权衡表现,同时独特地支持严格预算控制、解码期重路由以及单个训练模型内支持多种预算。

原文摘要 · Abstract (English)

Large language models (LLMs) incur high inference cost due to their depth and parameter scale. Depth pruning can reduce latency by skipping redundant Transformer blocks, but existing methods (i) provide limited control under user-specific compute budgets and (ii) typically fix the routing path, failing to adapt as the context grows during decoding. We propose Buddy, a budget-driven dynamic depth routing framework. Buddy uses a lightweight Decision Module to score intermediate layers conditioned on the input and deterministically executes the top-k layers to satisfy a given budget. To support decode-time adaptation, Buddy reuses the first-layer KV cache as a low-overhead global context source and pools it together with the newest token representation before each routing decision. When no explicit budget is provided, an optional Budget Predictor estimates an input-dependent compute level to balance quality and efficiency. Experiments on Llama-family and Qwen models show that Buddy is competitive with strong static pruning baselines and often improves the accuracy-compute trade-off, while uniquely supporting strict budget control, decode-time rerouting, and multiple budgets within a single trained model.

动态推理模型压缩预算控制LLM优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。