构建代码与文档协同更新数据集，助力软件维护中的智能编辑
CoDocBench: A Dataset for Code-Documentation Alignment in Software Maintenance
- 从真实GitHub项目中提取代码与注释同步变更的成对数据
- 现有大模型在代码-文档对齐任务上表现仍不理想
- 适合研究代码理解、自动修复与AI编程助手的开发者
软件维护的核心任务之一是理解并生成代码变更。给定函数新功能的自然语言描述，需生成对应代码修改；反之，给定代码修改，需生成新的功能说明。为此，我们构建了一个大规模、真实的代码-文档协同变更数据集CoDocBench，来自高质量GitHub项目，每个样本对应一次提交中代码与文档同时更新的实例。我们介绍了数据采集方法，并展示了若干具有挑战性但贴近实际的任务场景，可用于模型训练与评估。实验发现，当前主流模型（如Llama-3.1 405B、Mixtral 8×22B）在这些任务上仍面临显著困难。
原文摘要 · Abstract (English)
One of the central tasks in software maintenance is being able to understand and develop code changes. Thus, given a natural language description of the desired new operation of a function, an agent (human or AI) might be asked to generate the set of edits to that function to implement the desired new operation; likewise, given a set of edits to a function, an agent might be asked to generate a changed description, of that function's new workings. Thus, there is an incentive to train a neural model for change-related tasks. Motivated by this, we offer a new, "natural", large dataset of coupled changes to code and documentation mined from actual high-quality GitHub projects, where each sample represents a single commit where the code and the associated docstring were changed together. We present the methodology for gathering the dataset, and some sample, challenging (but realistic) tasks where our dataset provides opportunities for both learning and evaluation. We find that current models (specifically Llama-3.1 405B, Mixtral 8$\times$22B) do find these maintenance-related tasks challenging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。