构建高可信代码变更缺陷数据集，揭示大模型其实没真正理解代码改动。
ReDef: Do Code Language Models Truly Understand Code Changes for Just-in-Time Software Defect Prediction?
- 基于回滚提交构建3164个有缺陷、10268个无缺陷的函数级修改数据集
- 多种反事实扰动测试显示模型性能稳定，说明依赖表面特征而非语义理解
- 验证了现有代码大模型对代码变更的理解存在严重局限性
即时软件缺陷预测（JIT-SDP）在代码审查和持续集成中至关重要。然而，现有数据集常因标签噪声和缺陷定位不准而受限。为此，我们提出ReDef（基于回滚的缺陷数据集），从22个大型C/C++项目中构建高置信度的函数级修改数据集。缺陷样本由回滚提交锚定，清洁样本通过事后历史检查验证，模糊样本经GPT辅助多轮投票与审计过滤。该流程生成3,164个缺陷和10,268个清洁修改，显著优于以往资源。我们还系统评估了CodeBERT、CodeT5+、UniXcoder和Qwen2.5等代码语言模型对代码修改的推理能力。先考察五种输入编码策略对变化信息的表达效果，再设计四种反事实扰动（如交换增删块、反转diff极性）作为诊断探针。若模型真理解语义，扰动应导致性能明显下降。结果表明，紧凑的diff编码在所有模型上均优于全函数格式，且统计显著；但扰动下性能几乎不变，表明所谓‘鲁棒性’实为依赖表面线索，缺乏深层语义理解。
原文摘要 · Abstract (English)
Just-in-Time software defect prediction (JIT-SDP) plays a critical role in prioritizing risky code changes during code review and continuous integration. However, existing datasets often suffer from noisy labels and low precision in identifying bug-inducing commits. To address this, we present ReDef (Revert-based Defect dataset), a high-confidence benchmark of function-level modifications curated from 22 large-scale C/C++ projects. Defective cases are anchored by revert commits, while clean cases are validated through post-hoc history checks. Ambiguous instances are conservatively filtered out via a GPT-assisted triage process involving multiple votes and audits. This pipeline yields 3,164 defective and 10,268 clean modifications, offering substantially more reliable labels than prior resources. Beyond dataset construction, we provide a systematic evaluation of how Code Language Models (CLMs)-specifically CodeBERT, CodeT5+, UniXcoder, and Qwen2.5-reason about code modifications. We first investigate which input encodings most effectively expose change information under five different strategies. We then design four counterfactual perturbation strategies (e.g., swapping added/deleted blocks, inverting diff polarity) to serve as diagnostic probes. We posit that if models genuinely capture change semantics, such distortions should lead to a clear decline in predictive performance. Our results show that compact diff-style encodings consistently outperform whole-function formats across all CLMs, supported by rigorous statistical confirmation. However, under counterfactual tests, performance remains effectively stable, revealing that what appears to be robustness in fact reflects a reliance on superficial cues rather than true semantic understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。