首次实证研究代码修复中测试过拟合问题
Investigating Test Overfitting on SWE-bench
- 分析自动构建测试时的过拟合风险
- 发现修复代码在自动生成测试上表现好但实际功能差
- 警示依赖自动生成测试的修复系统需警惕过拟合
测试对解决代码仓库中的问题有帮助，但过度依赖测试可能导致代码仅通过表面测试却遗漏关键场景甚至破坏功能，这种现象称为测试过拟合。该问题因多数问题缺乏可执行测试而加剧。现有系统常从问题描述自动生成测试，这些测试可能存在缺陷；部分系统还迭代优化代码与测试。本文首次在该场景下开展实证研究，揭示测试过拟合的普遍性与严重性。
原文摘要 · Abstract (English)
Tests can be useful towards resolving issues on code repositories. However, relying too much on tests for issue resolution can lead to code that technically passes observed tests but actually misses important cases or even breaks functionality. This problem, called test overfitting, is exacerbated by the fact that issues usually lack readily executable tests. Instead, several issue resolution systems use tests auto-generated from issues, which may be imperfect. Some systems even iteratively refine code and tests jointly. This paper presents the first empirical study of test overfitting in this setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。