arXiv:2606.10935cs.LGcs.AI2026-06

用轻量级预测器实现零损失多标记推理加速,解决重复生成问题。

CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference

  • 让主干模型生成首词,多标记预测头只负责后续词,避免冲突
  • 仅用4.6K-7.7K参数的预测层,实现1.2倍加速且无质量下降
  • 适合追求高效推理的大型语言模型部署者

大语言模型推理受自回归解码限制,每生成一个词需完整前向传播。多标记预测(MTP)虽具加速潜力,但现有方法存在根本缺陷:首个词的MTP头与主干语言模型头竞争,导致接受预测时质量严重下降。我们识别出该头-主干竞争是先前MTP加速方法产生重复、不连贯输出的根本原因。为此提出「主干即架构」设计原则——主干模型始终生成首词,MTP头仅负责后续词。在此基础上,引入轻量级跨度级决策层CLP(共现长度预测器),预测每步可安全接受的额外词数。CLP仅使用单个线性层(4.6K–7.7K参数),取代此前过复杂(100万参数)的门控网络。在Qwen2.5模型(0.5B、1.5B、7B)上的实验表明,CLP在1.5B模型上实现1.20x–1.29x加速,在7B模型上实现1.14x–1.20x加速,且无质量损失(重复率<0.02);而门控方法要么无法加速(1.07x),要么输出严重劣化(重复率>0.5%)。进一步发现较短预测范围(k=2)在大模型上提升24%的MTP头准确率,确立了与规模适配的设计原则。我们指出MTP头预测准确率是加速的瓶颈,并为未来改进提供清晰路径。

原文摘要 · Abstract (English)

Large language model inference is bottlenecked by autoregressive decoding, where each token requires a full forward pass. Multi-token prediction (MTP) offers a promising acceleration path, but existing approaches suffer from a fundamental architectural flaw: the MTP head for the first token competes with the backbone's own language model (LM) head, leading to severe quality degradation when predictions are accepted. We identify this head-backbone competition as the root cause of repetitive and incoherent outputs in prior MTP-based acceleration methods. To address this, we propose Backbone-as-Architect, a design principle where the backbone LM head always generates the first token, and MTP heads are responsible only for subsequent tokens. Building on this principle, we introduce CLP (Collocation-Length Predictor), a lightweight span-level decision layer that predicts how many additional tokens can be safely accepted at each decoding step. CLP uses only a single linear layer (4.6K--7.7K parameters), replacing the over-engineered 1M-parameter gate networks used in prior work. Experiments on Qwen2.5 models (0.5B, 1.5B, 7B) show that CLP achieves 1.20x--1.29x speedup on 1.5B and 1.14x--1.20x on 7B, with zero quality degradation (repetition ratio < 0.02), while gate-based approaches fail to accelerate (1.07x) or produce severely degraded outputs (repetition ratio > 0.5%). We further demonstrate that shorter prediction horizons (k=2) recover 24% higher MTP head accuracy on large models, establishing a scaling-aware design principle. We identify MTP head prediction accuracy as the binding constraint on acceleration and establish a clear roadmap for future improvements.

多标记预测推理加速语言模型轻量设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。