用规则代码+大模型双重验证，提升指令跟随强化学习效果
VerIF: Verification Engineering for Reinforcement Learning in Instruction Following
- 结合规则检查与大模型推理进行双重验证
- 在2.2万条指令数据上训练，显著提升模型表现
- 适合想提升指令遵循能力的模型开发者
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型（LLMs）性能的关键技术，其中验证工程起核心作用。然而，指令跟随场景下的强化学习最佳实践仍不明确。本文探讨了该任务中的验证挑战，提出VerIF方法，融合规则驱动的代码验证与大型推理模型（如QwQ-32B）的LLM验证。为此，我们构建了一个高质量指令跟随数据集VerInstruct，包含约22,000个实例及对应的验证信号。将VerIF应用于两个模型进行强化学习训练，在多个代表性指令跟随基准上均取得显著提升，达到同规模模型中的最先进水平，并对未见约束具有良好泛化能力。进一步发现，模型通用能力未受影响，表明VerIF可无缝集成至现有强化学习流程以提升整体性能。相关数据集、代码与模型已开源：https://github.com/THU-KEG/VerIF。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing large language models (LLMs), with verification engineering playing a central role. However, best practices for RL in instruction following remain underexplored. In this work, we explore the verification challenge in RL for instruction following and propose VerIF, a verification method that combines rule-based code verification with LLM-based verification from a large reasoning model (e.g., QwQ-32B). To support this approach, we construct a high-quality instruction-following dataset, VerInstruct, containing approximately 22,000 instances with associated verification signals. We apply RL training with VerIF to two models, achieving significant improvements across several representative instruction-following benchmarks. The trained models reach state-of-the-art performance among models of comparable size and generalize well to unseen constraints. We further observe that their general capabilities remain unaffected, suggesting that RL with VerIF can be integrated into existing RL recipes to enhance overall model performance. We have released our datasets, codes, and models to facilitate future research at https://github.com/THU-KEG/VerIF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。