无需外部标注，直接从指令生成奖励信号，让模型更好理解复杂指令。
Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following
- 从指令中自动生成奖励信号，摆脱对外部标注依赖
- 通过约束分解与二分类策略，解决多约束任务奖励稀疏问题
- 在3个领域内和5个跨领域数据集上表现优异，适合复杂指令场景
语言模型在遵循多约束指令时表现不佳，而现有强化学习方法依赖外部监督且面临奖励信号稀疏的问题。本文提出一种无需标签的自监督强化学习框架，直接从指令中提取奖励信号，并生成伪标签用于奖励模型训练。通过引入约束分解策略和高效的约束级二分类机制，在保持计算效率的同时缓解奖励稀疏性。实验表明，该方法在3个领域内和5个跨领域数据集上均取得显著提升，涵盖具有挑战性的代理式和多轮指令跟随任务。代码与数据已公开于 https://github.com/Rainier-rq/verl-if。
原文摘要 · Abstract (English)
Language models often struggle to follow multi-constraint instructions that are crucial for real-world applications. Existing reinforcement learning (RL) approaches suffer from dependency on external supervision and sparse reward signals from multi-constraint tasks. We propose a label-free self-supervised RL framework that eliminates dependency on external supervision by deriving reward signals directly from instructions and generating pseudo-labels for reward model training. Our approach introduces constraint decomposition strategies and efficient constraint-wise binary classification to address sparse reward challenges while maintaining computational efficiency. Experiments show that our approach generalizes well, achieving strong improvements across 3 in-domain and 5 out-of-domain datasets, including challenging agentic and multi-turn instruction following. The data and code are publicly available at https://github.com/Rainier-rq/verl-if
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。