无需外部监督,通过语义熵和行为共识自我优化代码生成。
Self-Improving Code Generation via Semantic Entropy and Behavioral Consensus
- 用代码语义熵衡量问题难度,构建可学性高的学习课程。
- 基于行为一致性加权偏好对,减少自生成标注噪声影响。
- 在多个基准上超越基线,适合无教师模型的代码生成场景。
提升大语言模型的代码生成能力通常依赖于监督微调或偏好优化,但这两者都需要昂贵的外部资源,如强大的教师模型或可靠的测试用例。然而,在真实场景中,获取参考解和测试断言比获取问题描述和测试输入更难。本文探讨一个更具现实意义的问题:代码语言模型能否在没有优越教师模型和测试断言的情况下实现自我改进?为此,我们提出 ConSelf,一种基于两个核心思想的自提升方法。首先,引入代码语义熵,一种通过评估程序行为的功能多样性来衡量问题层面不确定性的新指标,从而构建最具可学性的学习课程。其次,提出基于共识的直接偏好优化(Con-DPO),一种按行为一致性加权每个偏好对的偏好优化方法,有效缓解自生成监督中的噪声影响。在多个基准和基础大模型上的实验表明,ConSelf 显著优于基线,验证了基于语义熵的课程构建与共识驱动优化在无外部监督下提升代码生成的有效性。
原文摘要 · Abstract (English)
Improving the code generation capabilities of large language models (LLMs) typically relies on supervised fine-tuning or preference optimization, both of which require costly external resources such as powerful teacher models or reliable test units. However, in real-world scenarios, it is much harder to obtain reference solutions and test oracles than problem descriptions and test inputs. In this paper, we tackle a challenging yet realistic question: Can a code language model improve itself without access to a superior teacher and a test oracle? To answer this, we propose ConSelf, a self-improving approach built upon two key ideas. First, we introduce code semantic entropy, a novel metric that measures problem-level uncertainty by assessing the functional diversity of program behaviors, enabling a curriculum construction with the most learnable problems. Second, we present consensus-driven direct preference optimization (Con-DPO), a preference-based fine-tuning method that weights each preference pair by its behavioral consensus, thereby mitigating the impact of noisy self-generated supervision. Experiments on various benchmarks and backbone LLMs demonstrate that ConSelf significantly outperforms baselines, validating the effectiveness of semantic entropy-based curriculum construction and consensus-driven optimization in improving code generation without external supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。