测试大模型对代码异常的鲁棒性，发现顶尖模型在扰动下准确率暴跌20%-24%
How Robustly do LLMs Understand Execution Semantics?
- 通过代码扰动测试模型对执行语义的理解能力
- GPT-5.2在扰动后准确率下降20%~24%，降至75%以下
- 异常类型影响预测效果，需针对性改进
大语言模型虽具备强大推理能力，但其是否依赖内部世界模型或仅靠模式匹配尚不明确。本文通过标准程序输出预测任务，从鲁棒性角度研究模型对代码执行语义的理解。结果揭示显著差异：开源推理模型（DeepSeek-R1系列）在代码变换与输入扰动下表现稳定，准确率维持在38%至67%；而前沿模型GPT-5.2在原始CRUXEval基准上达99%准确率，但在扰动输入下准确率下降20%至24%。此外，多数模型在引发异常的扰动输入上表现更差，且预测性能受异常类型影响。本文探讨了提升异常预测能力的修复方法，并评估其对非异常行为预测的影响。研究揭示了当前模型在代码理解上的普遍局限，也验证了扰动测试在评估代码模型中的价值。
原文摘要 · Abstract (English)
LLMs demonstrate remarkable reasoning capabilities, yet whether they utilize internal world models or rely on sophisticated pattern matching remains open. We study LLMs through the lens of robustness of their code understanding using a standard program-output prediction task. Our results reveal a stark divergence in model behavior: while open-source reasoning models (DeepSeek-R1 family) maintain stable, albeit somewhat lower accuracies (38% to 67%) under code transformations & input perturbations, the frontier model GPT-5.2 exhibits significant brittleness. Despite achieving a near-perfect score of 99% on the original, unperturbed CRUXEval benchmark, perturbed inputs trigger accuracy declines between 20% and 24%. In addition, we find that many models perform much worse at predicting behavior on perturbed inputs that raise exceptions, and that prediction performance depends on the kind of exception. We study remedies to address this deficiency in exception prediction, and evaluate the effect of these remedies on the ability to predict non-exception behaviors. Our findings both point to limitations in the way all models understand code, and establish the value of using perturbation to evaluate code models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。