评估大模型在非功能需求编码中的鲁棒性，发现同一需求不同表达会导致结果不一致。
RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation
- 通过提示词变化、回归测试和工作流对比，多角度评测大模型生成代码的稳定性。
- 加入非功能需求后，通过率下降39%，结果波动标准差从0.48升至2.48。
- 适合关注大模型代码生成可靠性的开发者与研究者参考。
当使用大语言模型（LLM）处理非功能需求（NFRs）时，开发者可能以不同方式表达相同需求。鲁棒的LLM应在这些变化下输出一致结果，但该问题尚未充分研究。本文提出RobuNFR，用于评估LLM在设计、可读性、可靠性与性能四个维度上，对非功能需求感知代码生成的鲁棒性，采用提示词变化、回归测试和多样化工作流三种方法。实验表明，引入非功能需求会使Pass@1下降最多39%，标准差从0.48增至2.48，且整体提升非功能指标的同时加剧了提示敏感性。在回归测试中，部分模型在版本间表现不一致，如减少代码异味常伴随正确性下降，暴露出不一致性。不同工作流下，初始提示整合与后期增强两种方式生成的代码质量差异显著，揭示了模型在任务流程上的敏感性。
原文摘要 · Abstract (English)
When using LLMs to address Non-Functional Requirements (NFRs), developers may behave differently (e.g., expressing the same NFR in different words). Robust LLMs should output consistent results across these variations; however, this aspect remains underexplored. We propose RobuNFR for evaluating the robustness of LLMs in NFR-aware code generation across four NFR dimensions: design, readability, reliability, and performance, using three methodologies: prompt variation, regression testing, and diverse workflows. Our experiments show that RobuNFR reveals robustness issues in the tested LLMs when considering NFRs in code generation. Specifically, under prompt variation, including NFRs leads to a decrease in Pass@1 by up to 39 percent and an increase in the standard deviation from 0.48 to 2.48 compared to the baseline without NFRs (i.e., Function-Only). While incorporating NFRs generally improves overall NFR metrics, it also results in higher prompt sensitivity. In regression settings, some LLMs exhibit differences across versions, with improvements in one aspect (e.g., reduced code smells) often accompanied by regressions in another (e.g., decreased correctness), revealing inconsistencies that challenge their robustness. When varying workflows, the tested LLMs show significantly different NFR-aware code generation capabilities between two workflows: (1) integrating NFRs and functional requirements into the initial prompt and (2) enhancing Function-Only-generated code with the same NFR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。