提出可精确控制生成长度的新型值模型,提升效率与性能。
Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
- 将长度建模转化为每步的值估计问题,用负奖励实现无标注监督。
- 在Qwen2.5上使精确长度得分从30.9提升至83.6,长文本任务保留63%正确率。
- 适用于大模型长度控制、预测及轨迹分析,支持持续优化与解释。
现代自回归模型以分词为计算单元,生成长度直接影响推理成本与推理表现。现有方法多在粗粒度序列层面建模长度。本文提出长度值模型(LenVM),在解码每一步预测剩余生成长度。通过将长度建模视为值估计问题,并为每个生成分词赋予恒定负奖励,LenVM输出有界、折扣化的回报,作为剩余生成周期的单调代理。该框架提供无标注、密集、无偏且可扩展的监督信号。在大语言模型和视觉语言模型上的实验表明,LenVM支持精确控制、连续性能-效率调节、长度预测与可解释性。在LIFEBench-token上,使Qwen2.5-7B-Instruct的精确长度得分从30.9提升至64.8(单次运行),结合LCG后达83.6。在GSM8K近200词场景下,相比硬预算限制,保持约63%的Pass@1,而后者仅6%。在其他设置中,其匹配长度质量优于提示工程与结束符校准,且长度预测准确率随规模增长,超过专用预测器。其分词级值信号还可识别长/短轨迹的转换标志。结果表明,生成长度是可用于控制、预测与未来基于值训练的有效可扩展分词级信号。
原文摘要 · Abstract (English)
Tokens are the fundamental units of computation in modern autoregressive models, and generation length directly influences both inference cost and reasoning performance. Despite its importance, existing approaches model length primarily at the coarse sequence level. We introduce the Length Value Model (LenVM), a token-level framework that estimates the remaining generation length at every decoding step. By formulating length modeling as a value estimation problem and assigning a constant negative reward to each generated token, LenVM predicts a bounded, discounted return that is a monotone proxy for the remaining generation horizon. This value formulation provides annotation-free, dense, unbiased, and scalable supervision. Experiments on LLMs and VLMs show that LenVM supports exact control, continuous performance--efficiency steering, length prediction, and interpretation. On LIFEBench-token, it raises the exact-length score of Qwen2.5-7B-Instruct from $30.9$ to $64.8$ in one pass and reaches $83.6$ when combined with LCG. Near $200$ tokens on GSM8K, LenVM retains about $63\%$ Pass@1 versus $6\%$ under a hard token budget. Across other settings, LenVM improves matched-length quality over prompting and EOS calibration, while its length-prediction accuracy improves with scale and exceeds specialized predictors. Its token-level values also identify markers of shifts between longer and shorter trajectories. These results establish generation length as an effective scalable token-level value signal for control, prediction, and future value-based training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。