arXiv:2605.30218cs.LGcs.PF2026-05被引 3

只对可能出错的步骤做验证,让大模型推理更稳定又更快。

MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference

  • 只在低置信度步骤触发验证,大幅减少计算开销。
  • 在多个数据集上实现100%推理确定性,验证触发率仅15%-50%。
  • 适合追求高稳定性和低延迟的大模型部署场景。

温度为零的BF16大模型推理常被视为可复现,但相同请求在单独解码与批量解码时可能产生不同结果。现有方法采用批量不变操作或逐标记验证,即使多数步骤稳定也需开销。本文发现:批量引起的标记翻转极为稀疏——在MATH500上,Llama-3.1-8B仅0.48%的同步解码步骤发生翻转,所有模型在MATH500、GSM8K和HumanEval上翻转率均保持在0.3%-1.3%之间。键值(K/V)扰动在翻转前保持平稳,而低top-1/top-2 logits差距暴露了大部分翻转风险。基于此,MarginGate提出一种验证策略:高置信度步骤继续使用BF16解码,仅对低置信度步骤进行验证,并通过替换当前K/V列修复确认的不一致。在四个数据集上评估,以MATH500校准后迁移至GSM8K、SharedGPT和HumanEval,该方法使Llama-3.1-8B和Qwen2.5-14B实现100%序列级确定性,验证触发率分别为18.56%和15.05%,相较始终开启验证,降低LLM-42延迟提升2.23倍/1.99倍。在更严苛的DSR1-Distill-Qwen-7B上,相同策略以49.50%触发率实现确定性。

原文摘要 · Abstract (English)

Temperature-zero BF16 LLM inference is often treated as reproducible, yet the same request can emit different tokens when decoded alone or inside a larger batch. Existing fixes use batch-invariant operators or LLM-42's per-token verification, incurring cost even when most steps are stable. We ask whether verification can be applied exclusively to flipped tokens. Across five models, batch-induced token flips are sparse on the flip-rate benchmarks: on MATH500, Llama-3.1-8B flips on $0.48\%$ of synchronous decode steps, and all tested models stay within the 0.3-1.3% range on MATH500, GSM8K, and HumanEval. K/V perturbations remain flat before flips, while low top-1/top-2 logit margins expose much of the flip risk. MarginGate turns these observations into a verifier policy: it keeps BF16 decoding on high-margin steps, verifies only low-margin steps, and repairs confirmed mismatches by replacing the current K/V column. We evaluate on four datasets, calibrating on MATH500 and transferring to GSM8K, SharedGPT, and HumanEval. MarginGate restores 100% sequence-level deterministic decoding on Llama-3.1-8B and Qwen2.5-14B with 18.56%/15.05% verifier trigger rates, reducing LLM-42's latency increment by 2.23x/1.99x relative to always-on verification. On DSR1-Distill-Qwen-7B, the same policy reaches determinism in a harder regime at 49.50% triggers.

大模型推理确定性优化验证机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。