大模型能自动生成用户故事并评估其质量,提升敏捷开发效率。
Can LLMs Generate User Stories and Assess Their Quality?
- 用10个顶尖大模型模拟客户访谈生成用户故事
- 生成的故事在覆盖度和风格上接近人类,但创意与多样性较低
- 在明确标准下可可靠评估语义质量,适合大规模需求审查
需求获取仍是需求工程中最具挑战性的环节,因分析人员难以将复杂需求转化为具体要求。高质量需求直接影响软件质量。尽管自动化工具可评估语法质量,但语义质量(如语言清晰度、内部一致性)仍需人工完成,耗时费力。本文研究大模型在敏捷框架中自动生成用户故事的能力,使用10个前沿大模型模拟客户访谈生成用户故事,并与领域专家及学生生成的结果进行对比。结果表明,大模型生成的用户故事在覆盖率和风格上与人类相当,但多样性与创造力较低;尽管整体质量接近人类,但满足验收标准的情况更少,且不随模型规模提升而改善。此外,当提供明确评价标准时,大模型能可靠评估用户故事的语义质量,有潜力显著降低大规模评估的人工成本。
原文摘要 · Abstract (English)
Requirements elicitation is still one of the most challenging activities of the requirements engineering process due to the difficulty requirements analysts face in understanding and translating complex needs into concrete requirements. In addition, specifying high-quality requirements is crucial, as it can directly impact the quality of the software to be developed. Although automated tools allow for assessing the syntactic quality of requirements, evaluating semantic metrics (e.g., language clarity, internal consistency) remains a manual and time-consuming activity. This paper explores how LLMs can help automate requirements elicitation within agile frameworks, where requirements are defined as user stories (US). We used 10 state-of-the-art LLMs to investigate their ability to generate US automatically by emulating customer interviews. We evaluated the quality of US generated by LLMs, comparing it with the quality of US generated by humans (domain experts and students). We also explored whether and how LLMs can be used to automatically evaluate the semantic quality of US. Our results indicate that LLMs can generate US similar to humans in terms of coverage and stylistic quality, but exhibit lower diversity and creativity. Although LLM-generated US are generally comparable in quality to those created by humans, they tend to meet the acceptance quality criteria less frequently, regardless of the scale of the LLM model. Finally, LLMs can reliably assess the semantic quality of US when provided with clear evaluation criteria and have the potential to reduce human effort in large-scale assessments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。