用RAG提升代码问答效果，比人类更全面但更啰嗦。
A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Case
- 用检索增强生成（RAG）改进LLM回答技术问题
- RAG回答比人类更全面（62.50%对54.17%）
- 适合开源项目减轻核心维护者负担
近年来，大型语言模型（LLMs）在软件开发支持中的应用持续增长，从辅助编码到提供对话式问答。本文与Mozilla基金会合作，评估了检索增强生成（RAG）在Mozilla Firefox项目中协助开发者的效果。通过对比真实开发者聊天室中的提问，分析了人类开发者、标准GPT模型和增强版GPT-RAG的响应。由Mozilla专家依据有用性、全面性和简洁性进行评估。结果显示，RAG响应的全面性优于人类（62.50%对54.17%），几乎同样有用（75.00%对79.17%），表明其在开源软件（OSS）中提升支持能力的潜力。但RAG响应不够简洁，常显冗长。未来优化检索机制并缩短输出，将有助于在像Firefox这样的大型项目中进一步提升开发协助效率。
原文摘要 · Abstract (English)
The use of Large Language Models (LLMs) to support tasks in software development has steadily increased over recent years. From assisting developers in coding activities to providing conversational agents that answer newcomers' questions. In collaboration with the Mozilla Foundation, this study evaluates the effectiveness of Retrieval-Augmented Generation (RAG) in assisting developers within the Mozilla Firefox project. We conducted an empirical analysis comparing responses from human developers, a standard GPT model, and a GPT model enhanced with RAG, using real queries from Mozilla's developer chat rooms. To ensure a rigorous evaluation, Mozilla experts assessed the responses based on helpfulness, comprehensiveness, and conciseness. The results show that RAG-assisted responses were more comprehensive than human developers (62.50% to 54.17%) and almost as helpful (75.00% to 79.17%), suggesting RAG's potential to enhance developer assistance. However, the RAG responses were not as concise and often verbose. The results show the potential to apply RAG-based tools to Open Source Software (OSS) to minimize the load to core maintainers without losing answer quality. Toning down retrieval mechanisms and making responses even shorter in the future would enhance developer assistance in massive projects like Mozilla Firefox.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。