探究提示细节对大模型代码生成的影响，发现更具体提示显著提升性能。
More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation
- 设计渐进式提示层级评估框架，量化提示详细程度的影响。
- 在HumanEval和ParEval上，提示越具体，通过率提升明显，最高达30%以上。
- 明确输入输出格式、边界情况处理和分步拆解是关键改进点，适合工具开发者参考。
当前最先进的大语言模型（LLMs）在通用基准HumanEval上表现良好，但在专业领域如ParEval上性能下降。这种差距是由于模型缺乏领域知识，还是提示信息不足？为此，我们提出PartialOrderEval，通过构建从最简到最详尽的提示层级序列，系统评估代码生成任务中提示具体性的影响。我们在Llama-3.x与Qwen2.5-Coder上测试了HumanEval及ParEval的串行与OpenMP子集，发现不同任务对提示敏感度各异；定性分析表明，明确的输入输出规范、边界情况处理和分步推理结构是提升生成质量的关键因素。
原文摘要 · Abstract (English)
State-of-the-art Large Language Models (LLMs) achieve high pass@1 on general benchmarks like HumanEval but underperform on specialized suites such as ParEval. Is this due to LLMs missing domain knowledge or insufficient prompt detail is given? To answer this, we introduce PartialOrderEval, which augments any code generation benchmark with a partial order of prompts from minimal to maximally detailed. Applying it to HumanEval and both serial and OpenMP subsets of ParEval, we measure how pass@1 scales with prompt specificity. Our experiments with Llama-3.x and Qwen2.5-Coder demonstrate varying degrees of prompt sensitivity across different tasks, and a qualitative analysis highlights explicit I/O specifications, edge-case handling, and stepwise breakdowns as the key drivers of prompt detail improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。