让自回归模型一次生成多个词元,提升速度不降质量。
MARS: Enabling Autoregressive Models Multi-Token Generation
- 轻量微调使模型单步预测多词元,无需改架构或加参数。
- 多词元生成时吞吐提升1.5-1.7倍,准确率保持基准水平。
- 支持动态调整生成速度,适合高并发部署场景。
自回归语言模型逐词元生成文本,即使后续词元高度可预测。我们提出MARS(Mask AutoRegreSsion),一种轻量级微调方法,使指令调优的自回归模型能在单次前向传播中预测多个词元。MARS无需架构修改或额外参数,生成方式与原始模型完全兼容,无性能损失。不同于需额外草稿模型的推测解码或多头结构如Medusa,MARS仅需在现有指令数据上继续训练即可。在六项标准基准测试中,单词元生成时,MARS表现持平或超越基线;允许每步生成多个词元时,保持基线准确率的同时实现1.5–1.7倍吞吐提升。我们进一步设计块级键值缓存策略,在Qwen2.5-7B上实现最高1.71倍的实时加速。此外,通过置信度阈值调节,系统可在高负载下动态提升吞吐,无需换模型或重启,为部署提供实用的延迟-质量调控机制。
原文摘要 · Abstract (English)
Autoregressive (AR) language models generate text one token at a time, even when consecutive tokens are highly predictable given earlier context. We introduce MARS (Mask AutoRegreSsion), a lightweight fine-tuning method that teaches an instruction-tuned AR model to predict multiple tokens per forward pass. MARS adds no architectural modifications, no extra parameters, and produces a single model that can still be called exactly like the original AR model with no performance degradation. Unlike speculative decoding, which maintains a separate draft model alongside the target, or multi-head approaches such as Medusa, which attach additional prediction heads, MARS requires only continued training on existing instruction data. When generating one token per forward pass, MARS matches or exceeds the AR baseline on six standard benchmarks. When allowed to accept multiple tokens per step, it maintains baseline-level accuracy while achieving 1.5-1.7x throughput. We further develop a block-level KV caching strategy for batch inference, achieving up to 1.71x wall-clock speedup over AR with KV cache on Qwen2.5-7B. Finally, MARS supports real-time speed adjustment via confidence thresholding: under high request load, the serving system can increase throughput on the fly without swapping models or restarting, providing a practical latency-quality knob for deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。