通过预判令牌重要性与可恢复性,实现更智能的推理计算节省。
Informed Routing in LLMs: Smarter Token-Level Computation for Faster Inference
- 引入轻量预测模块预估令牌输出,指导执行或近似决策
- 在多个稀疏度下达成顶尖效率与性能平衡,无需全量微调
- 训练时间减少超50%,适合高并发低延迟部署场景
大型语言模型在实际应用中的部署日益受限于高昂的推理成本。尽管近期动态令牌级计算分配技术尝试通过按令牌选择性激活模型组件来提升效率,但现有方法依赖贪婪路由——一种短视的执行或跳过机制,常导致不可逆的信息损失和次优的令牌选择。本文提出「知情路由」新范式,主动解决上述问题。核心思想是评估令牌的即时重要性及其可恢复性,即其变换能否被良好近似。为此,我们提出轻量特征预测器(LFF),一个小型预测模块,在路由决策前预估单元输出。这使得系统可采用灵活的执行或近似策略,在大幅降低计算量的同时保持模型保真度。在语言建模与推理任务上的大量实验表明,知情路由在多种稀疏度下均实现当前最优的效率-性能权衡。值得注意的是,即使不进行最终的LoRA微调,该方法仍能匹配或超越需全量微调的强基线,同时训练时间减少超过50%。代码已公开于:https://github.com/EIT-NLP/informed-routing
原文摘要 · Abstract (English)
The deployment of large language models (LLMs) in real-world applications is increasingly limited by their high inference cost. While recent advances in dynamic token-level computation allocation attempt to improve efficiency by selectively activating model components per token, existing methods rely on greedy routing--a myopic execute-or-skip mechanism that often leads to irreversible information loss and suboptimal token selection. This paper introduces informed routing, a new paradigm that proactively addresses these issues. The key insight is to assess not only a token's immediate importance but also its recoverability, i.e., how well its transformation can be approximated. To this end, we propose the Lightweight Feature Forecaster (LFF), a small predictive module that estimates a unit's output before routing decisions are made. This enables a flexible execute-or-approximate policy that preserves model fidelity while drastically reducing computation. Extensive experiments on both language modeling and reasoning tasks show that informed routing achieves state-of-the-art efficiency-performance trade-offs across multiple sparsity levels. Notably, even without final LoRA fine-tuning, our method matches or surpasses strong baselines that require full fine-tuning, all while reducing training time by over 50%. The code is available at: https://github.com/EIT-NLP/informed-routing
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。