用检索增强生成技术识别并解决需求文档中的语用歧义问题。
A Retrieval-Augmented Framework for Detecting and Resolving Pragmatic Ambiguities in Natural Language Requirements

- 通过不同层级知识库模拟不同专家,检测需求理解差异。
- GPT-4o-mini在歧义检测中召回率最高(0.75),效果最佳。
- 生成的需求澄清版本经人工评估,兼具相关性、清晰性和一致性。
自然语言需求(NLRs)在软件开发中对弥合利益相关者间沟通鸿沟至关重要,但其内在歧义可能带来挑战。某些需求因利益相关者上下文知识和领域期望差异而被误解,这种现象称为语用歧义。本文提出一种检测与解决语用歧义的方法,利用检索增强生成技术,结合新手、中级和专家级领域知识库,模拟不同专业水平的利益相关者,识别需求解读差异。使用专家知识库生成候选澄清需求,最终由需求分析师验证以确保符合预期功能。在公共需求数据集的两份需求文档上,采用四种大模型(GPT-4o-mini、Mistral-7B、Llama-3.1-8B、Qwen2.5-7B)进行评估。检测性能通过宏平均准确率、精确率、召回率、F1和F2分数衡量;分辨率通过人工评估相关性、清晰性和一致性。结果表明,该方法能有效检测语用歧义,并生成相关、清晰且一致的澄清需求。其中,GPT-4o-mini在检测任务中取得最高宏平均召回率(0.75)和F2分数(0.75);在解析任务中,其生成内容相关性得分最高,而Mistral-7B在清晰性和一致性方面表现最优。
原文摘要 · Abstract (English)
Natural language requirements (NLRs) are essential for bridging communication gaps among diverse stakeholders in software development. However, the inherent ambiguity in NLRs can pose significant challenges. In particular, some requirements may be misinterpreted due to varying contextual knowledge and domain-specific expectations of the stakeholders, a phenomenon known as pragmatic ambiguity. This paper presents an approach for detecting and resolving pragmatic ambiguities in NLRs. The approach leverages retrieval-augmented generation techniques with novice, intermediate, and expert domain knowledge bases to simulate stakeholders with varying domain expertise and detect discrepancies in requirement interpretation. Candidate disambiguated requirements are generated using the expert domain knowledge base, with final validation by a requirements analyst required to ensure alignment with the intended functionality. We evaluate the approach on two requirements specification documents from the PUblic REquirements dataset, using four large language models: GPT-4o-mini, Mistral-7B, Llama-3.1-8B, and Qwen2.5-7B. Detection performance is assessed using macro-averaged accuracy, precision, recall, F1, and F2 scores. The resolution quality of the candidate disambiguated requirements is measured through human evaluation of relevance, clarity, and consistency. In this initial evaluation, results show that the proposed approach can detect pragmatic ambiguities and produce candidate disambiguated requirements that are relevant, clear, and consistent with the intended system functionality. Among the evaluated models, GPT-4o-mini achieved the highest macro-averaged recall (0.75) and F2 score (0.75) for pragmatic ambiguity detection. In the resolution task, GPT-4o-mini received the highest relevance scores from human evaluators, while Mistral-7B achieved the highest scores for clarity and consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。