对比GPT与Gemini在需求工程中的表现，发现提示词设计影响大，多数任务仍落后于专业模型。
Generative Language Models Potential for Requirement Engineering Applications: Insights into Current Strengths and Limitations
- 用四个公开数据集测试GPT和Gemini在需求提取、分类等任务上的表现。
- 在命名实体识别中，两模型F1仅0.36~0.25，远低于最优模型的0.92。
- 问答任务表现较好，但其他任务均弱于专用机器学习模型，需精细提示工程。
传统语言模型已在软件工程领域广泛评估，但ChatGPT与Gemini的潜力尚未充分探索。本文通过综合案例研究，探讨两者在多种需求工程应用中的可行性。针对不同专家知识水平的提示词，分析其对模型预测准确率的影响。在4个公开基准数据集上比较了两模型与现有特定任务机器/深度学习模型及传统语言模型的表现。数据集包括Pure（7,445样本，需求提取）、PROMISE（622样本，需求分类）、REQuestA（300个问答对）和Aerospace（6347词，需求NER标注）。实验表明，相比ChatGPT，Gemini需更精细的提示工程才能获得准确结果。在需求提取任务中，最佳模型F1为0.86，而ChatGPT和Gemini分别达到0.76和0.77；需求分类任务最佳F1为0.96，两模型均为0.78；在命名实体识别任务中，最佳模型F1为0.92，而ChatGPT为0.36，Gemini仅为0.25；问答任务最佳F1为0.90，两模型分别达到0.91和0.88。结果显示，除了问答任务，两模型在其他任务上均未超越现有先进方法。
原文摘要 · Abstract (English)
Traditional language models have been extensively evaluated for software engineering domain, however the potential of ChatGPT and Gemini have not been fully explored. To fulfill this gap, the paper in hand presents a comprehensive case study to investigate the potential of both language models for development of diverse types of requirement engineering applications. It deeply explores impact of varying levels of expert knowledge prompts on the prediction accuracies of both language models. Across 4 different public benchmark datasets of requirement engineering tasks, it compares performance of both language models with existing task specific machine/deep learning predictors and traditional language models. Specifically, the paper utilizes 4 benchmark datasets; Pure (7,445 samples, requirements extraction),PROMISE (622 samples, requirements classification), REQuestA (300 question answer (QA) pairs) and Aerospace datasets (6347 words, requirements NER tagging). Our experiments reveal that, in comparison to ChatGPT, Gemini requires more careful prompt engineering to provide accurate predictions. Moreover, across requirement extraction benchmark dataset the state-of-the-art F1-score is 0.86 while ChatGPT and Gemini achieved 0.76 and 0.77,respectively. The State-of-the-art F1-score on requirements classification dataset is 0.96 and both language models 0.78. In name entity recognition (NER) task the state-of-the-art F1-score is 0.92 and ChatGPT managed to produce 0.36, and Gemini 0.25. Similarly, across question answering dataset the state-of-the-art F1-score is 0.90 and ChatGPT and Gemini managed to produce 0.91 and 0.88 respectively. Our experiments show that Gemini requires more precise prompt engineering than ChatGPT. Except for question-answering, both models under-perform compared to current state-of-the-art predictors across other tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。