用在线强化学习让AI写代码既安全又可用，解决旧方法顾此失彼的问题。
Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Model
- 通过漏洞诱导任务和推理型奖励模型，构建在线强化学习闭环。
- 在多个数据集上使有效安全率提升约10%，优于未对齐模型。
- 适合需要高安全性且不牺牲功能性的代码生成场景。
大型语言模型（LLMs）在软件开发中应用日益广泛，但其生成不安全代码的倾向仍是实际部署的主要障碍。现有安全对齐方法常陷入功能与安全的悖论，提升安全性时显著降低实用性。本文提出 SecCoderX，一种保持功能性的在线强化学习框架，用于生成安全代码。该框架通过两种方式连接漏洞检测与代码生成：(i) 构造多样且贴近现实的漏洞诱导编程任务，用于在线强化学习的采样；(ii) 训练基于推理的漏洞奖励模型，提供可扩展且可靠的安全部署监督。这两项组件整合进在线强化学习循环中，实现代码模型在保持功能性的同时生成安全代码。大量实验表明，SecCoderX 在有效安全率（ESR）上相比未对齐模型提升约10%，而先前方法通常使 ESR 下降14%至54%。代码、数据集与模型权重已开源于 https://github.com/AndrewWTY/SecCoderX。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used in software development, yet their tendency to generate insecure code remains a major barrier to real-world deployment. Existing secure code alignment methods often suffer from a functionality--security paradox, improving security at the cost of substantial utility degradation. We propose SecCoderX, an online reinforcement learning framework for functionality-preserving secure code generation. SecCoderX first bridges vulnerability detection and secure code generation by repurposing mature detection resources in two ways: (i) synthesizing diverse, reality-grounded vulnerability-inducing coding tasks for online RL rollouts, and (ii) training a reasoning-based vulnerability reward model that provides scalable and reliable security supervision. Together, these components are unified in an online RL loop to align code LLMs to generate secure and functional code. Extensive experiments demonstrate that SecCoderX achieves state-of-the-art performance, improving Effective Safety Rate (ESR) by approximately 10% over unaligned models, whereas prior methods often degrade ESR by 14-54%. We release our code, dataset and model checkpoints at https://github.com/AndrewWTY/SecCoderX.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。