测试大模型在代码推理中被误导语言干扰的脆弱性，发现普遍依赖表面提示。
CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
- 构建1279个问题的测试框架，注入误导性自然语言干扰
- 平均性能下降23.2%，即使使用思维链也仍有13.8%降幅
- 揭示模型易陷入错误反思，导致推理崩溃，适合研究鲁棒性者参考
大型语言模型（LLMs）在代码任务中表现出强大能力，但其在结构扰动和误导性自然语言（NL）环境下推理的可靠性尚未充分探索。我们提出CodeCrash，一个包含1,279个问题的应力测试框架，源自CruxEval和LiveCodeBench，用于评估模型在误导性语言上下文下的推理稳健性。对17个LLMs的系统评估显示，模型常通过过度依赖自然语言线索进行推理捷径，导致输出预测任务平均性能下降23.2%。即使采用思维链（Chain-of-Thought）方法，仍存在平均13.8%的性能下降，反映出模型在辨别真实代码行为方面缺乏批判性推理能力。尽管具备内部推理机制的大模型能提升稳健性，但看似合理却错误的提示仍会引发病态自我反思，使QwQ-32B等模型出现2-3倍的令牌消耗，极端情况下甚至导致认知失调。我们称此现象为‘推理崩溃’（Reasoning Collapse）。CodeCrash为代码推理鲁棒性提供了严谨基准，可引导未来研究向更可靠、更健壮的模型发展。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have recently demonstrated strong capabilities in code-related tasks, but their robustness in code reasoning under perturbations remains underexplored. We introduce CodeCrash, a stress-testing framework with 1,279 questions from CruxEval and LiveCodeBench, designed to evaluate reasoning reliability under structural perturbations and misleading natural language (NL) contexts. Through a systematic evaluation of 17 LLMs, we find that models often shortcut reasoning by over-relying on NL cues, leading to an average performance degradation of 23.2% in output prediction tasks. Even with Chain-of-Thought reasoning, models on average still have a 13.8% drop due to distractibility and rationalization, revealing a lack of critical reasoning capability to distinguish the actual code behaviors. While Large Reasoning Models with internal reasoning mechanisms improve robustness by fostering critical thinking, plausible yet incorrect hints can trigger pathological self-reflection, causing 2-3 times token consumption and even catastrophic cognitive dissonance in extreme cases for QwQ-32B. We refer to this phenomenon as Reasoning Collapse. CodeCrash provides a rigorous benchmark for evaluating robustness in code reasoning, guiding future research and development toward more reliable and resilient models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。