AI生成的代码重构在真实项目中表现好坏参半,需加强质量与安全审查。
Quality and Security Signals in AI-Generated Python Refactoring Pull Requests

- 用机器学习工具与静态分析结合评估AI重构代码的质量和安全
- 22.5%的修改提升质量,36.5%改善可用性,但24.17%引入新代码规范问题
- 73.5%的重构请求被合并,即使引入新安全或规范问题
随着AI代理参与代码开发与维护,其在真实项目中重构行为的质量与风险特征仍缺乏实证研究。本文基于AIDev数据集,对Python重构类拉取请求(PR)开展实证分析。采用基于机器学习的PyQu工具量化五项代码质量属性变化,并结合Pylint和Bandit进行通用静态分析,评估变更前后代码质量与安全问题。结果表明,平均22.5%的变更提升了质量属性,其中可用性改善最频繁(36.5%)。同时,24.17%的文件引入新Pylint问题,主要为编码规范违规(如长行);4.7%引入新Bandit发现。通过对差异内容分析,归纳出24种常见修改操作,并映射其最常引发的警告类型。尽管存在混合结果,开发者接受度高:73.5%的PR被合并,且常伴随旧问题清除。研究揭示了智能重构的潜力与局限,呼吁在AI驱动开发流程中引入更强的工具链质量与安全拦截机制。
原文摘要 · Abstract (English)
As AI agents increasingly contribute to code development and maintenance, there is still limited empirical evidence on the quality and risk characteristics of their changes in real-world projects, particularly for refactoring-oriented contributions. It remains unclear how agent-authored refactoring edits affect maintainability, code quality, and security once merged into GitHub repositories. To address this gap, we conduct an empirical study of Python refactoring pull requests (PRs) from the AIDev dataset. We analyze agentic refactoring PRs using PyQu, an ML-based quality assessment tool for Python, to quantify changes across five quality attributes, and we complement PyQu with domain-independent static analysis (Pylint and Bandit) to measure code quality and security issues before and after each change. Our results show that, on average, agentic commits improve a quality attribute in 22.5% of the studied changes, with usability improving most frequently (36.5%). At the same time, 24.17% of modified files introduce new Pylint issues predominantly convention level violations such as long lines-while 4.7% introduce new Bandit findings. From the observed diffs, we derive a taxonomy of 24 recurring change operations and map them to the lint and security findings they most commonly affect. Despite these mixed outcomes, developer acceptance is high: 73.5% of the analyzed PRs are merged, including cases that introduce new lint or security findings, often alongside the removal of existing issues. Overall, these findings highlight both the promise and current limitations of agentic refactoring, and motivate stronger tool-in-the-loop quality and security gating for AI-driven development workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。