发现重喂会引入噪声，影响对语言模型决策关键词的因果评估准确性。
Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation
- 通过三步实验对比重喂与精确状态恢复，验证重喂带来的偏差
- 在低置信度决策点，重喂导致信用估计误差达14-28个百分点
- 建议使用状态恢复或批处理不变内核，避免因重喂引入误判
基于逐词反事实信用评估的框架中，传统方法通过重喂前缀来模拟生成状态。本文通过三阶段设计：从真实解码时的键值状态续接、完全相同的第二轮复现（噪声基准）、以及重喂重构。在六种配置和三个模型（含GRPO训练检查点）下，低置信度决策词处，重喂引入的信用估计偏差比复现基准高出14-28个百分点（7-21个百分点在无处理依赖条件下），统计显著（问题聚类t=2.9-6.4）。多数偏差源于量化估计器的零边界穿越，而非极性反转；平均量级仍可信赖，但选择结果不可靠：基于|Âₜ|阈值选出的关键词集与精确续接结果的交并比仅0.34-0.90，低于复现基准的0.63-0.96。因果验证显示，在vLLM的批处理不变内核下，三轮完全一致，分歧率为零。复现本身在9-23%的估计中存在不一致：单样本信用测量在决策词上本就不稳定。实验提前设定，第二轮缓存命中率达100%（3,434个枢轴点），总计算成本低于10美元。建议采用解码器状态续接或批处理不变内核，并报告复现噪声基准。
原文摘要 · Abstract (English)
Per-token counterfactual credit estimation asks which token in a language-model rollout caused the final answer to be right or wrong: cut the transcript at a pivot, substitute an alternative token, replay continuations, and compare outcomes. Published methods re-feed the transcript prefix as a fresh prompt, assuming this reproduces the state the model passed through during generation. We measure what that assumption costs on a stock inference engine, with a three-pass design: continuations resumed from the verified decode-time KV state, an identical second exact pass (a replica noise floor), and a re-feed pass. Across six configurations and three models (including a GRPO-trained checkpoint), at low-margin decision tokens, re-feeding changes the credit estimate at rates 14-28 percentage points above the replica floor (7-21pp under a treatment-independent conditioning; problem-clustered t = 2.9-6.4). Most changes are zero-boundary crossings of the quantized estimator rather than polarity reversals, and the perturbation is consistent with mean-zero, so averaged quantities are largely safe; but selection is not: a critical-token set chosen by thresholding $|\hat{A}_t|$ under re-feed overlaps the exact-resume selection at Jaccard 0.34-0.90, versus a 0.63-0.96 replica ceiling. A causal confirmation closes the loop: under vLLM's batch-invariant kernels all three passes are identical on every measured channel, with both disagreement rates exactly zero. Replica passes themselves disagree on 9-23% of eligible estimates: single-sample credit measurements at decision tokens are unreliable under any replay. Settings were fixed in advance; exact-pass cache hits in the second campaign are instrumented (100% hit rate, 3,434 pivots); total compute was under 10 USD. We recommend that counterfactual credit studies resume decoder state or use batch-invariant kernels, and report a replica floor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。