用GPT-5分析代码变更影响，数据集与实验揭示其潜力有限
A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis
- 构建含种子变更与影响实体的代码变更数据集
- GPT-5表现优于GPT-5-mini，但整体准确率仍低
- 差值片段（diff hunk）微弱提升模型性能，适合工具研发者参考
理解源码变更及其对其他代码实体的影响是软件开发中的关键能力。然而，当前分析多依赖人工，耗时费力。近年来大语言模型（LLMs）在代码分析任务中展现潜力，但其在代码变更影响分析中的应用仍不明确。为此，我们研究GPT-5与GPT-5-mini预测代码变更影响实体的能力。构建了一个包含种子变更、变更对及变更类型的提交数据集，弥补现有数据集缺乏种子变更与影响实体信息的缺陷。实验评估两种配置：(1) 种子变更信息与父提交树；(2) 增加每个种子变更的diff hunks。结果表明，两类模型表现均不佳，但GPT-5优于GPT-5-mini；提供diff hunks可使两者性能略有提升。
原文摘要 · Abstract (English)
Understanding source code changes and their impact on other code entities is a crucial skill in software development. However, the analysis of code changes and their impact is often performed manually and therefore is time-consuming. Recent advancements in AI, and in particular large language models (LLMs) show promises to help developers in various code analysis tasks. However, the extent to which this potential can be utilized for understanding code changes and their impact is underexplored. To address this gap, we study the capabilities of GPT-5 and GPT-5-mini to predict the code entities impacted by given source code changes. We construct a dataset containing information about seed-changes, change pairs, and change types for each commit. Existing datasets lack crucial information about seed changes and impacted code entities. Our experiments evaluate the LLMs in two configurations: (1) seed-change information and the parent commit tree and (2) seed-change information, the parent commit tree, and the diff hunk of each seed change. We found that both LLMs perform poorly in the two experiments, whereas GPT-5 outperforms GPT-5-mini. Furthermore, the provision of the diff hunks helps both models to slightly improve their performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。