用精确终态验证揭露代码生成中的执行陷阱
QuoteBench: How Matched Scores Can Hide Command-Path Failures

- 设计新基准测试,通过故意未转义的解析器暴露生成与执行间的断层
- 相同回复在不同执行路径下成功率下降55.4至73.2个百分点
- 揭示模型真实能力需看配置与执行路径,非单一匹配分数
LLM代码代理通过可能序列化、包装和重新解析输出的接口发出Bash命令。仅靠匹配执行分数无法区分生成错误与生成后引入的失败。QuoteBench在14个事故衍生任务族的56个单次任务上,通过在故意未转义的解析器处跨过生成契约,实现精确终态验证。在插入选项点进行转义可重现每条重播回复的原始路径结果,因此任何在披露边界下的恢复必来自模型生成改变。在八个同窗口配置中,通过新增解析器重播相同回复,成功率下降55.4至73.2个百分点;披露后六个配置恢复30.4至60.7个百分点,其余两个则零或轻微负向。原始生成几乎逼近前沿;边界适应才是模型间真正分界。GPT-5.6-sol的匹配差距-3.6点隐藏了-64.3点损伤与+60.7点补偿。部署配置使模型排序发生逆转:26组可比对中有一次明确反转,四组位于单任务边缘。命令发出代理的评估应报告模型配置、生成契约、执行路径、运行点及终态验证器,而非将匹配分数视为模型固有属性。
原文摘要 · Abstract (English)
LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。