提出首个代码去污染工具包,有效降低模型评估中的数据泄露风险。
CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkit
- 设计11种代码重构操作,覆盖方法、类及跨类层级
- 应用所有操作后数据重叠率下降65%,显著提升评估可信度
- 开源工具支持Python与Java,适合工业界落地使用
数据污染是阻碍先进代码语言模型(CLMs)在工业界广泛应用的关键障碍。当评估数据意外与训练所用公开代码库重叠时,会严重削弱性能评估的可信度。对于考虑将基于CLM的技术引入开发流程的软件企业而言,真实性能指标的不确定性构成不可接受的商业风险。代码重构(包括结构调整和变量重命名)已成为一种有前景的缓解手段,可替代耗时耗力的无污染数据集构建。然而,缺乏自动化重构工具和科学验证的重构技术,制约了其工业应用。本文首次系统研究多尺度(方法级、类级、跨类级)及多语言下的重构操作有效性,开发开源工具CODECLEANER,包含11种针对Python的重构操作(9种方法级、1种类级、1种跨类级)。实验显示,应用全部操作后数据重叠率下降65%。此外,将其中4个操作迁移至Java,验证其泛化能力。CODECLEANER已公开,以推动对CLM数据污染问题的进一步研究。
原文摘要 · Abstract (English)
Data contamination presents a critical barrier preventing widespread industrial adoption of advanced software engineering techniques that leverage code language models (CLMs). This phenomenon occurs when evaluation data inadvertently overlaps with the public code repositories used to train CLMs, severely undermining the credibility of performance evaluations. For software companies considering the integration of CLM-based techniques into their development pipeline, this uncertainty about true performance metrics poses an unacceptable business risk. Code refactoring, which comprises code restructuring and variable renaming, has emerged as a promising measure to mitigate data contamination. It provides a practical alternative to the resource-intensive process of building contamination-free evaluation datasets, which would require companies to collect, clean, and label code created after the CLMs' training cutoff dates. However, the lack of automated code refactoring tools and scientifically validated refactoring techniques has hampered widespread industrial implementation. To bridge the gap, this paper presents the first systematic study to examine the efficacy of code refactoring operators at multiple scales (method-level, class-level, and cross-class level) and in different programming languages. In particular, we develop an open-sourced toolkit, CODECLEANER, which includes 11 operators for Python, with nine method-level, one class-level, and one cross-class-level operator. A drop of 65% overlap ratio is found when applying all operators in CODECLEANER, demonstrating their effectiveness in addressing data contamination. Additionally, we migrate four operators to Java, showing their generalizability to another language. We make CODECLEANER online available to facilitate further studies on mitigating CLM data contamination.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。