用自洽循环测试大模型编程鲁棒性，发现性能强不等于抗错能力强。
EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programming
- 让模型在代码与自然语言间反复转换，直到出错为止，评估其自我一致性。
- 提出平均可持续循环数（ASL）指标，96个模型测试中准确率下降2.65%至47.62%。
- 适合关注AI编程工具长期稳定性的研究者和开发者使用。
评估大语言模型（LLMs）在编程任务中的鲁棒性对保障AI辅助软件开发的可靠性至关重要。然而，传统对抗攻击存在根本性局限：不同攻击策略导致评价结果矛盾，且仅依赖外部扰动，无法反映自主编码代理在内部生成后续输入时的内在稳定性。为此，本文提出EVALOOOP框架，从自洽性视角评估鲁棒性，利用软件工程任务天然的双重性（如代码生成与代码摘要）。该框架构建自包含反馈环，使LLM在代码与自然语言间迭代转换直至功能失效，鲁棒性由新提出的平均可持续循环数（ASL）度量——即在基准任务中保持功能正确的平均迭代次数。此循环策略无需外部攻击配置，提供统一指标，揭示模型在持续自指变换中维持语义完整的能力。我们在含MBPP Plus基准的EVALOOOP上评估了96个主流模型（参数量0.5B至685B），发现十轮内平均准确率下降2.65%至47.62%。有趣的是，初始性能并非鲁棒性指标：例如Qwen3-235B-A22B-Instruct-2507虽初始生成能力弱于o系列和DeepSeek-V3，但表现出更高鲁棒性（最高ASL得分）。
原文摘要 · Abstract (English)
Evaluating the programming robustness of large language models (LLMs) is paramount for ensuring their reliability in AI-based software development. However, adversarial attacks exhibit fundamental limitations that compromise fair robustness assessment: they demonstrate contradictory evaluation outcomes where different attack strategies tend to favor different models, and more critically, they operate solely through external perturbations, failing to capture the intrinsic stability essential for autonomous coding agents where subsequent inputs are endogenously generated by the model itself. We introduce EVALOOOP, a novel assessment framework that evaluates robustness from a self-consistency perspective, leveraging the natural duality inherent in software engineering tasks (e.g., code generation and code summarization). EVALOOOP establishes a self-contained feedback loop where an LLM iteratively transforms between code and natural language until functional failure occurs, with robustness quantified by a novel Average Sustainable Loops (ASL) metric-the mean number of iterations maintaining functional correctness across benchmark tasks. This cyclical strategy intrinsically evaluates robustness without relying on external attack configurations, providing a unified metric that reveals how effectively LLMs preserve semantic integrity through sustained self-referential transformations. We evaluate 96 popular LLMs, ranging from 0.5B to 685B parameters, on EVALOOOP equipped with the MBPP Plus benchmark, and found that EVALOOOP typically induces a 2.65%-47.62% absolute drop in pass@1 accuracy within ten loops. Intriguingly, robustness does not always align with initial performance (i.e., one-time query); for instance, Qwen3-235B-A22B-Instruct-2507, despite inferior initial code generation compared to OpenAI's o-series models and DeepSeek-V3, demonstrated the superior robustness (ASL score).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。