用代码片段提升编程大模型,无需人工指令标注。
CodeBoost: Boosting Code LLMs by Squeezing Knowledge from Code Snippets with RL
- 从海量代码片段中自动筛选优质训练数据,构建多样化语料库。
- 通过正向与反向预测、错误反馈等机制,显著提升模型生成准确率。
- 适合希望低成本优化代码生成模型的研究者与开发者使用。
代码大语言模型已成为构建高效自动化编码流程的关键工具。现有模型通常基于强化学习从通用大模型微调,依赖人工标注的“人类指令-最终答案”对,但高质量代码指令的收集成本高且难扩展。相比之下,代码片段来源丰富。这种不平衡成为基于指令微调的主要瓶颈。我们提出 CodeBoost,一种完全基于代码片段的微调框架,无需依赖人工标注指令。其核心包括:(1) 最大团筛选,从代码中选取具有代表性和多样性的训练语料;(2) 双向预测,使模型同时学习正向与反向生成目标;(3) 错误感知预测,利用正确与错误输出共同提供学习信号;(4) 异构增强,通过多样化数据扩充提升代码语义表达;(5) 异构奖励,结合格式正确性及执行成功/失败反馈进行多维度引导。在多个代码大模型和基准测试上实验表明,CodeBoost持续提升性能,验证了其作为可扩展、高效的训练流水线的有效性。
原文摘要 · Abstract (English)
Code large language models (LLMs) have become indispensable tools for building efficient and automated coding pipelines. Existing models are typically post-trained using reinforcement learning (RL) from general-purpose LLMs using "human instruction-final answer" pairs, where the instructions are usually from manual annotations. However, collecting high-quality coding instructions is both labor-intensive and difficult to scale. On the other hand, code snippets are abundantly available from various sources. This imbalance presents a major bottleneck in instruction-based post-training. We propose CodeBoost, a post-training framework that enhances code LLMs purely from code snippets, without relying on human-annotated instructions. CodeBoost introduces the following key components: (1) maximum-clique curation, which selects a representative and diverse training corpus from code; (2) bi-directional prediction, which enables the model to learn from both forward and backward prediction objectives; (3) error-aware prediction, which incorporates learning signals from both correct and incorrect outputs; (4) heterogeneous augmentation, which diversifies the training distribution to enrich code semantics; and (5) heterogeneous rewarding, which guides model learning through multiple reward types including format correctness and execution feedback from both successes and failures. Extensive experiments across several code LLMs and benchmarks verify that CodeBoost consistently improves performance, demonstrating its effectiveness as a scalable and effective training pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。