用大模型提升代码与文档的关联能力，效果显著但需人工把关。
Evaluating the Use of LLMs for Documentation to Code Traceability
- 对比三款大模型在文档与代码间建立链接的能力
- 最佳模型F1达79.4%~80.4%，远超传统方法
- 适合做自动化追溯的开发人员和工具设计者
大语言模型（LLMs）为自动化文档到代码的追溯带来了新可能，但其能力仍待深入探索。本文系统评估了Claude 3.5 Sonnet、GPT-4o和o3-mini在连接软件文档（包括API参考和用户指南）与源代码方面的表现。基于两个开源项目（Unity Catalog和Crawl4AI）构建了两个新数据集。实验评估了三项核心能力：（1）追溯链接识别准确率，（2）关系解释质量，（3）多步链路重建能力。结果表明，最优模型在两个数据集上的F1分数分别达到79.4%和80.4%，显著优于基线方法（TF-IDF、BM25、CodeBERT）。尽管完全正确的解释比例为42.9%至71.1%，但部分正确率超过97%，说明基本关联极少遗漏。对于多步链路，模型在端点准确性上表现良好，但中间节点捕捉差异较大。错误分析显示，大量误报源于命名假设、虚假链接或架构模式过度泛化。研究证明任务设计（如一对多匹配策略）对性能至关重要。这表明LLMs可作为追溯发现的强大助手，但其局限性要求未来工具采用人机协同设计，并关注特定错误模式。
原文摘要 · Abstract (English)
Large Language Models (LLMs) offer new potential for automating documentation-to-code traceability, yet their capabilities remain underexplored. We present a comprehensive evaluation of LLMs (Claude 3.5 Sonnet, GPT-4o, and o3-mini) in establishing trace links between various software documentation (including API references and user guides) and source code. We create two novel datasets from two open-source projects (Unity Catalog and Crawl4AI). Through systematic experiments, we assess three key capabilities: (1) trace link identification accuracy, (2) relationship explanation quality, and (3) multi-step chain reconstruction. Results show that the best-performing LLM achieves F1-scores of 79.4% and 80.4% across the two datasets, substantially outperforming our baselines (TF-IDF, BM25, and CodeBERT). While fully correct relationship explanations range from 42.9% to 71.1%, partial accuracy exceeds 97%, indicating that fundamental connections are rarely missed. For multi-step chains, LLMs maintain high endpoint accuracy but vary in capturing precise intermediate links. Error analysis reveals that many false positives stem from naming-based assumptions, phantom links, or overgeneralization of architectural patterns. We demonstrate that task-framing, such as a one-to-many matching strategy, is critical for performance. These findings position LLMs as powerful assistants for trace discovery, but their limitations could necessitate human-in-the-loop tool design and highlight specific error patterns for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。