用免费的暂停令牌提升语言模型预测能力，不增加推理开销。
Free Pause Tokens
- 通过共享骨干网络的并行计算流实现额外推理，不增加序列长度。
- 在10亿参数模型上提升2-3个厘分，且推理延迟几乎不变。
- 训练成本仅增加1.14倍，适合高效部署的模型优化场景。
自由暂停令牌（Free Pause Token）为语言模型提供额外计算资源以生成下一个词的预测（如同暂停或思考令牌），但将该计算置于共享权重的并行预测流中，而非序列中的额外标记。这在10亿参数模型上实际提升了2-3个厘分的下一个词预测性能。由于暂停令牌使用已有位置而非新增序列位置，因此推理时无额外上下文长度、无键值缓存开销，且延迟基本不变；尽管推理浮点运算量增加，但通常不影响吞吐量瓶颈。训练阶段的主要成本仅为预训练流水线的1.14倍，同时保留了大部分优势。最终实现了与标准Transformer在相同浮点数、参数量和序列长度下的性能提升。
原文摘要 · Abstract (English)
A free pause token gives a language model extra compute to form each next-token prediction (as a pause, or thinking, token does) but carries that compute in a parallel prediction stream over a weight-shared backbone rather than as an extra token in the sequence. It improves next-token prediction by 2-3 centinats in practice on a 1B parameter model. Because the pause rides an existing position instead of adding one, it is free to use: at inference it adds no context length, no KV cache, and essentially no latency with the growth in inference flops typically irrelevant as it is not the active bottleneck on throughput. The only primary cost is in training, where additional training compute versus an optimized pretraining pipeline is reduced to as low as x1.14 while preserving most of the benefits. The result is an isoflop, isoparameter, and isotoken improvement over standard next token trained transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。