用RAG自动估算软件需求点,效果尚可但未显著优于人工。
Agile Story-Point Estimation: Is RAG a Better Way to Go?
- 用检索增强生成技术自动估算开发任务的复杂度
- 在23个开源项目上测试,准确率与传统方法无显著差异
- 适合希望减少手动估测时间的敏捷开发团队
敏捷开发中的冲刺规划会话需要团队对任务进行共识式估算,常用单位为故事点(Story Point),用于衡量任务复杂度和所需工时。然而这一过程高度依赖人工,耗时较长。本研究探索是否可用检索增强生成(RAG)技术实现自动化估算,RAG包含检索器与生成器两部分。研究使用bge-large-en-v1.5和Sentence-Transformers的all-mpnet-base-v2两个嵌入模型,在23个不同规模的开源项目上评估了四个关键问题:1)检索超参数对性能的影响;2)项目规模对估算准确率的影响;3)嵌入模型选择的影响;4)RAG方法与现有基线的对比。结果显示,尽管在某些情况下RAG优于基线模型,但整体性能在项目间及模型间均无统计学显著差异。这表明当前RAG在自动估算用户故事方面仍有改进空间,需进一步研究和模型适配策略以提升准确性。
原文摘要 · Abstract (English)
The sprint-based iterative approach in the Agile software development method allows continuous feedback and adaptation. One of the crucial Agile software development activities is the sprint planning session where developers estimate the effort required to complete tasks through a consensus-based estimation technique such as Planning Poker. In the Agile software development method, a common unit of measuring development effort is Story Point (SP) which is assigned to tasks to understand the complexity and development time needed to complete them. Despite the benefits of this process, it is an extremely time-consuming manual process. To mitigate this issue, in this study, we investigated if this manual process can be automated using Retrieval Augmented Generation (RAG) which comprises a "Retriever" and a "Generator". We applied two embedding models - bge-large-en-v1.5, and Sentence-Transformers' all-mpnet-base-v2 on 23 open-source software projects of varying sizes and examined four key aspects: 1) how retrieval hyper-parameters influence the performance, 2) whether estimation accuracy differs across different sizes of the projects, 3) whether embedding model choice affects accuracy, and 4) how the RAG-based approach compares to the existing baselines. Although the RAG-based approach outperformed the baseline models in several occasions, our results did not exhibit statistically significant differences in performance across the projects or across the embedding models. This highlights the need for further studies and refinement of the RAG, and model adaptation strategies for better accuracy in automatically estimating user stories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。