发现批量推测解码存在隐蔽错误，提出新方法提升速度与正确性。
Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problem
- 通过分析发现批量推测解码因张量不规则导致位置对齐错乱
- 新方法EXSPEC在批大小8时提速3倍，精确匹配率95%
- 适合关注推理效率与输出质量的模型部署开发者
推理优化常仅以吞吐量评估，却未验证输出正确性。我们对批量推测解码进行溯源分析，发现多个广泛使用的实现会无声产生错误输出（重复标记、<unk>符号），同时报告高吞吐，而此类错误无法被ROUGE等指标捕捉。根源在于张量不规则问题：不同序列的令牌接受情况导致位置ID、注意力掩码和KV缓存跨批次不同步。我们形式化了同步不变性（矩形对齐与位置ID连续性），并证明在连续布局下保持这些不变性会带来超线性对齐开销。EQSPEC无需自定义核函数即可强制不变性；EXSPEC将相同长度序列分组调度以绕过重对齐。在SpecBench上对三个模型族测试，EXSPEC在批大小8时达到3倍吞吐，且95%精确匹配标准解码；剩余差异源于浮点非确定性，而非同步错误。代码开源：https://github.com/eBay/spec_dec
原文摘要 · Abstract (English)
Inference optimizations are routinely evaluated by throughput alone, without verifying output correctness. We conduct a forensic analysis of batch speculative decoding and find that several widely-used implementations silently produce corrupted outputs (repetitive tokens, <unk> symbols) while reporting competitive speed; failures invisible to metrics like ROUGE. We trace the root cause to the ragged tensor problem: variable token acceptance desynchronizes position IDs, attention masks, and KV-cache across a batch. We formalize the synchronization invariants (rectangular alignment and position-ID contiguity) that valid batched inference must preserve and show that maintaining them incurs superlinear alignment overhead under contiguous layouts. EQSPEC enforces the invariants without custom kernels; EXSPEC schedules same-length sequences to bypass realignment. On SpecBench across three model families, EXSPEC reaches 3 x throughput at batch size 8 with 95% exact match to standard decoding; residual divergence traces to floating-point non-determinism, not synchronization error. Code:https://github.com/eBay/spec_dec
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。