用经验回放提升大模型代码生成效率，减少无效尝试。
Enhancing LLM Agents for Code Generation with Possibility and Pass-rate Prioritized Experience Replay
- 引入优先级回放机制，重用失败代码以加速学习。
- 在多个模型上验证，显著提升代码通过率与生成效率。
- 适合需要高效代码生成的开发工具或自动化系统使用。
当前基于Transformer的大语言模型在代码生成任务中通常采用采样与过滤流水线。由于代码生成中单个词错误即导致稀疏奖励，模型需反复采样冗余程序直至找到正确结果，效率低下。为此，我们在微调阶段引入经验回放（ER），将生成的代码与程序存储并重放，使模型能从过往经验中学习。提出BTP流水线，包含三阶段：束搜索采样、测试阶段和优先级经验回放阶段。该方法利用模型收集的失败程序，从回放缓冲区中优先重放具有高可能性与通过率优先值（P2Value）的程序。P2Value综合考虑输出可能性与通过率，有效利用了因多数程序无法通过测试而产生的冗余资源。在多个LLM上的实证表明，该方法显著提升了代码生成性能，优于现有基线。
原文摘要 · Abstract (English)
Nowadays transformer-based Large Language Models (LLM) for code generation tasks usually apply sampling and filtering pipelines. Due to the sparse reward problem in code generation tasks caused by one-token incorrectness, transformer-based models will sample redundant programs till they find a correct one, leading to low efficiency. To overcome the challenge, we incorporate Experience Replay (ER) in the fine-tuning phase, where codes and programs produced are stored and will be replayed to give the LLM agent a chance to learn from past experiences. Based on the spirit of ER, we introduce a novel approach called BTP pipeline which consists of three phases: beam search sampling, testing phase, and prioritized experience replay phase. The approach makes use of failed programs collected by code models and replays programs with high Possibility and Pass-rate Prioritized value (P2Value) from the replay buffer to improve efficiency. P2Value comprehensively considers the possibility of transformers' output and pass rate and can make use of the redundant resources caused by the problem that most programs collected by LLMs fail to pass any tests. We empirically apply our approach in several LLMs, demonstrating that it enhances their performance in code generation tasks and surpasses existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。