优化提示词格式可提升大模型编程效率，但自动调优可能适得其反。
PromptResponse: Optimizing Prompts for LLM Coding Tasks
- 对比五种提示格式，发现JSON格式最稳定高效。
- 使用GPT-4o在8200次执行中测试，JSON格式生成效率最高。
- 自动化提示调优导致任务性能显著下降，需谨慎使用。
大型语言模型（LLMs）在科研和开发流程中应用日益广泛，但其输出对提示词变化敏感。本文提出《PromptResponse》，通过受控实验研究编码任务提示词的格式与基于LLM的优化对其生成代码性能、效率和稳定性的影响。利用五个语义相同但语法不同的HumanEval数据集变体——基准版、JSON、Markdown、YAML以及一个由LLM优化的版本——我们让GPT-4o执行了8200次编码任务。结果表明，一致的格式（尤其是JSON）能显著提升生成效率和语法稳定性，仅带来微小的任务性能提升；而基于LLM的提示调优则导致任务性能明显下降，其他维度未见显著改善。研究建议：低成本重排格式可带来实际收益，但提示调优需考虑模型对齐问题。最后，我们提供了基于结果的实用建议，并公开了数据集变体与评估流水线以供后续研究。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used in research workflows and software development pipelines, yet their output remains sensitive to input prompt variations. This paper presents $\unicode{x00AB}$PromptResponse$\unicode{x00BB}$, a controlled study examining how formatting and LLM-based tuning of coding task prompts affect the resulting code's performance, efficiency, and stability. Using five semantically identical yet syntactically distinct variants of the HumanEval dataset$\unicode{x2014}$baseline, JSON, Markdown, YAML, and an LLM-tuned version$\unicode{x2014}$we had GPT-4o solve its coding problems over 8200$\unicode{x00A0}$executions. Our results show that consistent formatting$\unicode{x2014}$especially JSON$\unicode{x2014}$improves generation efficiency and syntactic stability, with minor gains in task performance. Conversely, the LLM-tuned prompts resulted in significantly degraded task performance without significant improvements in any other dimension. These findings suggest that low-effort reformatting alone can yield measurable improvements, while tuning must account for model alignment. We conclude our work with providing a set of practical recommendations informed by our results as well as releasing our dataset variants and evaluation pipeline for future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。