提出Cactus方法，在保持输出质量的同时提升自回归解码速度。
Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
- 通过约束优化框架设计接受策略，平衡速度与分布一致性。
- 在多个基准测试中实现更高吞吐量，且输出质量无明显下降。
- 适合需要高效推理但又要求生成质量的场景，如实时对话系统。
推测采样（SpS）通过使用较小的草稿模型加速自回归大语言模型的解码吞吐量。传统SpS严格要求生成分布与验证器大模型一致，过于保守。典型接受采样（TAS）虽通过熵启发式接受更多令牌以缓解此问题，但会扭曲验证器分布，可能在验证器编码关键信息时降低输出质量。本文从约束优化视角形式化推测采样算法，提出Cactus（约束接受推测采样），在保证与验证器分布可控偏差的同时提升接受率。大量实验证明该方法有效。
原文摘要 · Abstract (English)
Speculative sampling (SpS) has been successful in accelerating the decoding throughput of auto-regressive large language models by leveraging smaller draft models. SpS strictly enforces the generated distribution to match that of the verifier LLM. This is unnecessarily restrictive as slight variations of the verifier's distribution, such as sampling with top-$k$ or temperature, would also be acceptable. Typical acceptance sampling (TAS) alleviates this issue by accepting more tokens using entropy-based heuristics. However, this approach distorts the verifier distribution, potentially degrading output quality when the verifier encodes critical information. In this work, we formalize the speculative sampling algorithm through the lens of constrained optimization. Based on this formulation, we propose Cactus (constrained acceptance speculative sampling), a method that guarantees controlled divergence from the verifier distribution and increasing acceptance rates. Empirical results across a wide range of benchmarks confirm the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。