研究提示词变化对大模型代码生成质量的影响
Code Roulette: How Prompt Variability Affects LLM Code Generation
- 设计无任务依赖的提示增强评估流程
- 发现代码质量随提示微调显著波动
- 适合关注提示工程与生成稳定性的研究者
代码生成是大语言模型（LLMs）最活跃的应用领域之一。尽管LLMs降低了编程门槛并加速开发进程，但生成代码的功能性与质量仍高度依赖输入提示的质量。具体而言，生成结果对用户背景及软件开发熟悉度敏感。因此，量化LLM对提示变化的敏感性至关重要。为此，我们提出一种面向代码生成的评估流水线，专注于测量提示增强带来的影响，该方法不依赖特定编程任务或模型，具有广泛适用性。我们通过大量实验验证了该方法的有效性，并公开代码以促进社区发展。
原文摘要 · Abstract (English)
Code generation is one of the most active areas of application of Large Language Models (LLMs). While LLMs lower barriers to writing code and accelerate development process, the overall quality of generated programs depends on the quality of given prompts. Specifically, functionality and quality of generated code can be sensitive to user's background and familiarity with software development. It is therefore important to quantify LLM's sensitivity to variations in the input. To this end we propose an evaluation pipeline for LLM code generation with a focus on measuring sensitivity to prompt augmentations, completely agnostic to a specific programming tasks and LLMs, and thus widely applicable. We provide extensive experimental evidence illustrating utility of our method and share our code for the benefit of the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。