首个评估大模型修复架构异味的基准，发现其修复效果有限且易引入新问题。
SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair
- 构建专用提示与多步执行框架，系统评估大模型修复跨模块设计缺陷的能力。
- 最佳模型仅修复47.7%的异味，且最激进策略引入140个新异味。
- 适合关注AI辅助软件工程、代码质量提升的研究者与开发者。
架构异味降低软件可维护性，修复成本高，且需跨模块理解设计意图，对开发者和自动化工具均具挑战。尽管大语言模型在修复局部错误和代码重构方面表现优异，但其在修复架构异味方面的能力尚未被探索。本文提出首个针对该任务的实证评估，构建SmellBench框架，包含针对不同异味类型的优化提示和迭代多步执行支持，并采用独立评估修复有效性、误报识别与代码库净影响的评分方法。我们在Python项目scikit-learn中，基于PyExamine检测出的65个高严重性架构异味上，评估了来自GPT、Claude、Gemini、Mistral四个模型家族的11种代理配置，结果经专家判断验证。专家验证显示63.1%的检测结果为误报，最佳代理的修复率为47.7%。代理对误报的识别可达κ=0.94的专家一致性，但修复激进性与代码库整体质量呈负相关：最激进的代理引入140个新异味。这些发现揭示当前大模型在局部代码变换与所需架构理解之间存在差距。SmellBench为追踪自动化软件工程中这一未充分探索维度提供了可复用基础设施。代码与数据已公开于https://doi.org/10.5281/zenodo.19247588。
原文摘要 · Abstract (English)
Architectural code smells erode software maintainability and are costly to repair manually, yet unlike localized bugs, they require cross-module reasoning about design intent that challenges both developers and automated tools. While large language model agents excel at bug fixing and code-level refactoring, their ability to repair architectural code smells remains unexplored. We present the first empirical evaluation of LLM agents on architectural code smell repair. We contribute SmellBench, a task orchestration framework that incorporates smell-type-specific optimized prompts and supports iterative multi-step execution, together with a scoring methodology that separately evaluates repair effectiveness, false positive identification, and net codebase impact. We evaluate 11 agent configurations from four model families (GPT, Claude, Gemini, Mistral) on 65 hard-severity architectural smells detected by PyExamine in the Python project scikit-learn, validated against expert judgments. Expert validation reveals that 63.1% of detected smells are false positives, while the best agent achieves a 47.7% resolution rate. Agents identify false positives with up to $κ= 0.94$ expert agreement, but repair aggressiveness and net codebase quality are inversely related: the most aggressive agent introduces 140 new smells. These findings expose a gap between current LLM capabilities in localized code transformations and the architectural understanding needed for cross-module refactoring. SmellBench provides reusable infrastructure for tracking progress on this underexplored dimension of automated software engineering. We release our code and data at https://doi.org/10.5281/zenodo.19247588.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。