arXiv:2602.14012cs.CRcs.AI2026-02被引 2

首次系统分析大模型漏洞检测后训练流程,发现强化学习优于微调。

From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection

  • 用拒绝采样微调替代理性化监督,减少幻觉。
  • 基于根因判断的奖励机制避免奖励黑客,提升检测准确率。
  • 适合安全研究者和开发人员参考模型优化策略。

将大语言模型(LLM)引入漏洞检测(VD)使分析更可解释、上下文敏感。尽管后训练技术在通用编程任务中表现良好,但在漏洞检测中的系统应用仍不充分。本文首次全面研究基于大模型的漏洞检测后训练流程,结果表明:基于GRPO的在线强化学习(on-policy RL)始终优于微调(SFT)、离线偏好优化方法及专用漏洞检测模型。研究揭示了若干特定于漏洞检测的后训练准则:(1)数据筛选方面,与以往依赖理性化监督不同,基于拒绝采样的微调更有效;而强化学习中漏洞难度分布本就倾斜,导致难度感知过滤大幅降低数据覆盖,造成显著性能损失,并破坏课程学习,仅配对调度可部分缓解。(2)阶段交互上,相较于偏好优化通常用于轻度微调模型,增加微调轮次反而持续提升离线偏好优化表现;但过度微调会抑制在线强化学习中的自我探索,限制其收益。(3)奖励机制方面,直接以分类正确性为奖励信号易引发奖励黑客;而细粒度根因判断提供更可靠信用分配,基于规范的奖励进一步提升效率,但需额外设计与生成成本。(4)评估协议上,基于根因分析的LLM作为裁判(LLM-as-a-Judge)更具鲁棒性,尽管不同裁判模型间存在差异。

原文摘要 · Abstract (English)

The integration of LLMs into vulnerability detection (VD) has shifted the field toward more interpretable and context-aware analysis. While post-training techniques have shown promise in general coding tasks, their systematic application to VD remains underexplored. In this paper, we present the first comprehensive investigation into the post-training pipeline for LLM-based VD, demonstrating that on-policy RL with GRPO consistently outperforms SFT, off-policy preference optimization methods, and specialized VD LLMs. Our study further reveals VD-specific post-training guidelines and insights beyond common practices: (1) For data curation, contrary to the widespread use of rationalization-based supervision in prior VD work, SFT based on rejection sampling proves more effective, as rationalization can introduce hallucinations; in RL training, the inherently skewed difficulty distribution of vulnerabilities leads difficulty-aware data filtering to drastically reduce data coverage, causing non-negligible performance loss, and undermines curriculum learning, while pair-based data scheduling can partially mitigate this. (2) For stage interactions, unlike preference optimization typically applied to lightly trained SFT models, increasing SFT epochs consistently benefits off-policy preference optimization in VD tasks; however, excessive SFT suppresses self-exploration in on-policy RL, limiting its gains. (3) For reward mechanisms, naively treating vulnerability classification correctness as reward signals leads to reward hacking, whereas fine-grained root-cause judgments provide more reliable credit assignment; specification-based rewards further improve efficiency at the cost of additional design and generation effort. (4) For evaluation protocols, LLM-as-a-Judge based on root-cause analysis offers a more robust alternative, albeit with variability across judge models.

漏洞检测强化学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。