用知识图谱增强大模型，让代码库问答更准更快
Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering
- 用代码库数据构建知识图谱，与大模型协同理解问题
- 准确率提升至84%，显著优于基线方法
- 适合需要快速理解代码库的开发者和研究者
软件仓库蕴含丰富的开发过程信息，但提取其中洞察耗时且需技术专长。现有软件工程聊天机器人虽支持自然语言交互，却难以理解训练范围外的问题，且检索精度不足。本研究通过两步法提升基于大模型的聊天机器人在代码库相关问题上的准确性：首先从仓库数据构建知识图谱，再将其与大模型协同以处理自然语言问答。我们整理了150个不同复杂度的问题，在五个热门开源项目上评估该方法。初步结果显示，主要错误源于大模型的推理能力；为此引入少样本思维链提示（few-shot chain-of-thought prompting），使准确率提升至84%。与基准模型MSRBot和GPT-4o-search-preview相比，本方法表现更优。20名用户的任务型实验表明，使用本方法能更高效、更准确地完成任务，且用户反馈其具有实用价值。结果证明，大模型与知识图谱的结合是提升代码库可访问性的有效方案。
原文摘要 · Abstract (English)
Software repositories contain valuable information for understanding the development process. However, extracting insights from repository data is time-consuming and requires technical expertise. While software engineering chatbots support natural language interactions with repositories, chatbots struggle to understand questions beyond their trained intents and to accurately retrieve the relevant data. This study aims to improve the accuracy of LLM-based chatbots in answering repository-related questions by augmenting them with knowledge graphs. We use a two-step approach: constructing a knowledge graph from repository data, and synergizing the knowledge graph with an LLM to handle natural language questions and answers. We curated 150 questions of varying complexity and evaluated the approach on five popular open-source projects. Our initial results revealed the limitations of the approach, with most errors due to the reasoning ability of the LLM. We therefore applied few-shot chain-of-thought prompting, which improved accuracy to 84%. We also compared against baselines (MSRBot and GPT-4o-search-preview), and our approach performed significantly better. In a task-based user study with 20 participants, users completed more tasks correctly and in less time with our approach, and they reported that it was useful. Our findings demonstrate that LLMs and knowledge graphs are a viable solution for making repository data accessible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。