探究大模型在多轮编程中推理的机制，发现有效提示策略可提升性能。
What Makes Large Language Models Reason in (Multi-Turn) Code Generation?
- 通过分解提示类型，系统测试多轮自动重提示策略
- 在CodeContests和TACO基准上，所有模型性能均提升，小采样预算下也有效
- 微调后模型内化推理过程，多轮代码生成更高效可靠
提示技术如思维链已在大语言模型输出优化中广泛应用。然而，其在代码生成中的具体机制与效果仍不明确。本文系统研究多种提示策略，聚焦多轮自动重提示及计算开销。通过分解推理、指令与执行反馈提示，在竞赛编程基准CodeContests和TACO上对Llama 3.0/3.1（8B、70B、405B）及GPT-4o等多模型进行大规模网格搜索。结果表明，某些策略在所有模型上均显著提升性能，且在小采样预算下依然有效。进一步发现，采用最优配置微调后，模型可内化推理过程，在多轮代码生成中实现性能与可扩展性双重提升。
原文摘要 · Abstract (English)
Prompting techniques such as chain-of-thought have established themselves as a popular vehicle for improving the outputs of large language models (LLMs). For code generation, however, their exact mechanics and efficacy are under-explored. We thus investigate the effects of a wide range of prompting strategies with a focus on automatic re-prompting over multiple turns and computational requirements. After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o). Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets. We then show how finetuning with such an optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。