用模型内部状态预测输出长度，减少推理时的浪费。
Predicting LLM Output Length via Entropy-Guided Representations
- 复用主模型隐藏状态，通过熵引导池化实现高效长度预估。
- 在长序列数据集上，平均绝对误差比基线降低29.16%。
- 适合需要高效推理的场景，如强化学习采样和批量服务。
大型语言模型服务与强化学习采样中，序列长度呈现长尾分布，导致批处理推理时过度填充造成显著计算浪费。现有方法依赖辅助模型进行静态长度预测，但开销大、泛化差，且难以应对随机的“一到多”生成场景。本文提出一种轻量级框架，复用主模型内部隐藏状态实现高效长度预测。框架包含两个核心组件：1）熵引导令牌池化（EGTP），利用实时激活和令牌熵实现高精度静态预测，成本可忽略；2）渐进式长度预测（PLP），在解码每一步动态估计剩余长度，以应对随机生成。为验证方法，我们构建并发布ForeLen——一个涵盖长序列、思维链和强化学习数据的综合性基准。在ForeLen上，EGTP达到当前最优精度，相较最佳基线将平均绝对误差（MAE）降低29.16%。将该方法与长度感知调度器结合，显著提升端到端吞吐量。本工作为高效大模型推理提供了新的技术与评估基准。
原文摘要 · Abstract (English)
The long-tailed distribution of sequence lengths in LLM serving and reinforcement learning (RL) sampling causes significant computational waste due to excessive padding in batched inference. Existing methods rely on auxiliary models for static length prediction, but they incur high overhead, generalize poorly, and fail in stochastic "one-to-many" sampling scenarios. We introduce a lightweight framework that reuses the main model's internal hidden states for efficient length prediction. Our framework features two core components: 1) Entropy-Guided Token Pooling (EGTP), which uses on-the-fly activations and token entropy for highly accurate static prediction with negligible cost, and 2) Progressive Length Prediction (PLP), which dynamically estimates the remaining length at each decoding step to handle stochastic generation. To validate our approach, we build and release ForeLen, a comprehensive benchmark with long-sequence, Chain-of-Thought, and RL data. On ForeLen, EGTP achieves state-of-the-art accuracy, reducing MAE by 29.16\% over the best baseline. Integrating our methods with a length-aware scheduler yields significant end-to-end throughput gains. Our work provides a new technical and evaluation baseline for efficient LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。