用代码翻译训练大模型,提升多语言编程理解能力。
On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding
- 结合在线与离线强化学习,通过单元测试奖励引导代码翻译。
- 提出分组等价偏好优化,让模型区分代码功能等价性。
- 适合研究多语言代码生成与模型泛化能力的学者。
大型语言模型(LLMs)在代码生成任务中表现优异,但在主流编程语言(如Python、C++)与其他语言间仍存在显著性能差距。为缩小这一差距,我们利用代码翻译任务训练LLMs,实现编程能力跨语言迁移。为此提出OORL框架,融合在线与离线强化学习策略。在代码翻译过程中,采用基于单元测试的规则奖励信号指导在线强化学习。同时引入分组等价偏好优化(GEPO),通过中间表示(IRs)分组训练模型,使其能识别与源代码功能等价的表示,并利用组内表示间的等价关系信号。该机制使模型更深入理解代码功能细节。大量实验表明,使用代码翻译任务结合OORL训练的模型,在多个编程语言的代码基准上均取得显著性能提升。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve remarkable performance in code generation tasks. However, a significant performance disparity persists between popular programming languages (e.g., Python, C++) and others. To address this capability gap, we leverage the code translation task to train LLMs, thereby facilitating the transfer of coding proficiency across diverse programming languages. Moreover, we introduce OORL for training, a novel reinforcement learning (RL) framework that integrates on-policy and off-policy strategies. Within OORL, on-policy RL is applied during code translation, guided by a rule-based reward signal derived from unit tests. Complementing this coarse-grained rule-based reward, we propose Group Equivalent Preference Optimization (GEPO), a novel preference optimization method. Specifically, GEPO trains the LLM using intermediate representations (IRs) groups. LLMs can be guided to discern IRs equivalent to the source code from inequivalent ones, while also utilizing signals about the mutual equivalence between IRs within the group. This process allows LLMs to capture nuanced aspects of code functionality. By employing OORL for training with code translation tasks, LLMs improve their recognition of code functionality and their understanding of the relationships between code implemented in different languages. Extensive experiments demonstrate that our OORL for LLMs training with code translation tasks achieves significant performance improvements on code benchmarks across multiple programming languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。