用问答任务评估代码库文档质量,更真实反映其实际价值。
Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development

- 通过LLM理解文档并实现功能来评估文档质量。
- 构建4170个问答条目,涵盖功能识别、定位与实现完整度。
- 发现高质量文档可提升开发工具解决率20%,适合开发者参考。
软件文档对代码库理解至关重要。尽管大语言模型(LLMs)已能从代码片段生成文档,但现有基准存在两大缺陷:(1)缺乏整体性、代码库级别的评估;(2)依赖不可靠的评估方法,如以LLM为评判者,其标准模糊且缺乏代码库级知识。为此,我们提出SWD-Bench,一个新型代码库级别文档评估基准。受文档驱动开发启发,该方法通过评估LLM能否基于文档理解并实现功能来衡量文档质量,采用功能驱动的问答(QA)任务。基准包含三个相互关联的任务:(1)功能检测,判断功能是否被描述;(2)功能定位,评估相关文件定位准确性;(3)功能完成,衡量实现细节的完整性。我们通过挖掘高质量拉取请求,并融入代码库上下文,构建了含4170个条目的基准。实验揭示当前文档生成方法的局限性,表明源代码具有互补价值。值得注意的是,表现最佳方法生成的文档使SWE-Agent的问题解决率提升20.00%,证明高质量文档在支持文档驱动开发中的实际价值。
原文摘要 · Abstract (English)
Software documentation is crucial for repository comprehension. While Large Language Models (LLMs) advance documentation generation from code snippets to entire repositories, existing benchmarks have two key limitations: (1) they lack a holistic, repository-level assessment, and (2) they rely on unreliable evaluation strategies, such as LLM-as-a-judge, which suffers from vague criteria and limited repository-level knowledge. To address these issues, we introduce SWD-Bench, a novel benchmark for evaluating repository-level software documentation. Inspired by documentation-driven development, our strategy evaluates documentation quality by assessing an LLM's ability to understand and implement functionalities using the documentation, rather than by directly scoring it. This is measured through function-driven Question Answering (QA) tasks. SWD-Bench comprises three interconnected QA tasks: (1) Functionality Detection, to determine if a functionality is described; (2) Functionality Localization, to evaluate the accuracy of locating related files; and (3) Functionality Completion, to measure the comprehensiveness of implementation details. We construct the benchmark, containing 4,170 entries, by mining high-quality Pull Requests and enriching them with repository-level context. Experiments reveal limitations in current documentation generation methods and show that source code provides complementary value. Notably, documentation from the best-performing method improves the issue-solving rate of SWE-Agent by 20.00%, which demonstrates the practical value of high-quality documentation in supporting documentation-driven development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。