测试大模型代码生成对描述变化的敏感度，发现复杂任务更易出错。
Operational Robustness of LLMs on Code Generation
- 提出场景域分析法，量化描述微变导致错误所需的最小改动
- 四模型排序：Gemini-pro最佳，复杂任务和高级主题下鲁棒性下降
- 适用于评估代码生成模型在真实开发中的稳定性，适合工程团队参考
大型语言模型（LLMs）在软件开发中广泛用于生成程序代码，评估其鲁棒性至关重要。现有方法不适用于代码生成，因自然语言描述空间为离散。本文提出场景域分析法，旨在确定使LLMs产生错误输出所需的语言描述最小变化量。我们形式化证明了该方法的理论性质，并对四种前沿模型（Gemini-pro、Codex、Llama2、Falcon 7B）进行了广泛实验，成功将其鲁棒性从高到低排序。研究还发现，任务主题越复杂、样本解越高级（如多线程、数据结构），模型鲁棒性越低。
原文摘要 · Abstract (English)
It is now common practice in software development for large language models (LLMs) to be used to generate program code. It is desirable to evaluate the robustness of LLMs for this usage. This paper is concerned in particular with how sensitive LLMs are to variations in descriptions of the coding tasks. However, existing techniques for evaluating this robustness are unsuitable for code generation because the input data space of natural language descriptions is discrete. To address this problem, we propose a robustness evaluation method called scenario domain analysis, which aims to find the expected minimal change in the natural language descriptions of coding tasks that would cause the LLMs to produce incorrect outputs. We have formally proved the theoretical properties of the method and also conducted extensive experiments to evaluate the robustness of four state-of-the-art art LLMs: Gemini-pro, Codex, Llamma2 and Falcon 7B, and have found that we are able to rank these with confidence from best to worst. Moreover, we have also studied how robustness varies in different scenarios, including the variations with the topic of the coding task and with the complexity of its sample solution, and found that robustness is lower for more complex tasks and also lower for more advanced topics, such as multi-threading and data structures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。