用双层记忆增强代码生成,让AI更懂用户意图。
FALCON: Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization system
- 双层记忆架构:长期记忆存知识,短期记忆抓即时反馈。
- 在MBPP和Humaneval上分别领先4.5%和6.1%。
- 适合需要精准对齐用户需求的自动化编程场景。
大型语言模型在自动化代码生成方面取得显著进展,但在编码场景中仍难以对齐用户意图,主要受限于数据集多样性不足、无法覆盖特殊任务或边界情况,且监督微调与基于人类反馈的强化学习存在局限。为此,我们提出反馈驱动的自适应长短时记忆强化编码优化系统(FALCON)。该系统采用双层结构:全局层面利用长期记忆保留并应用已学知识以提升代码质量;局部层面通过短期记忆整合编译器与AI系统的即时反馈。此外,引入基于反馈奖励的元强化学习,解决全局-局部双层次优化问题,增强模型在多样化任务中的适应能力。大量实验表明,该方法在MBPP基准上较其他强化学习方法提升超过4.5个百分点,在Humaneval上提升6.1个百分点,达到当前最优性能。开源代码已公开于https://github.com/titurte/FALCON。
原文摘要 · Abstract (English)
Recently, large language models (LLMs) have achieved significant progress in automated code generation. Despite their strong instruction-following capabilities, these models frequently struggled to align with user intent in coding scenarios. In particular, they were hampered by datasets that lacked diversity and failed to address specialized tasks or edge cases. Furthermore, challenges in supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) led to failures in generating precise, human-intent-aligned code. To tackle these challenges and improve the code generation performance for automated programming systems, we propose Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization (i.e., FALCON). FALCON is structured into two hierarchical levels. From the global level, long-term memory improves code quality by retaining and applying learned knowledge. At the local level, short-term memory allows for the incorporation of immediate feedback from compilers and AI systems. Additionally, we introduce meta-reinforcement learning with feedback rewards to solve the global-local bi-level optimization problem and enhance the model's adaptability across diverse code generation tasks. Extensive experiments demonstrate that our technique achieves state-of-the-art performance, leading other reinforcement learning methods by more than 4.5 percentage points on the MBPP benchmark and 6.1 percentage points on the Humaneval benchmark. The open-sourced code is publicly available at https://github.com/titurte/FALCON.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。