用自监督强化学习提升推理模型的指令遵循能力。
Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following
- 利用模型内部信号实现无外部监督的自监督强化学习
- 指令遵循能力显著提升，推理性能保持不变
- 适合追求低成本、可扩展的模型优化研究者
推理模型在复杂问题求解上表现优异，但其推理能力与指令遵循能力之间存在显著权衡。现有提升指令遵循的方法依赖更强的外部模型，导致方法瓶颈和成本上升、访问受限等实际问题。本文提出一种自监督强化学习框架，通过利用推理模型自身的内部信号来改进指令遵循能力，无需外部监督。大量实验表明，该框架在保持推理性能的同时显著提升指令遵循能力，提供了一种可扩展且成本低廉的优化路径。数据与代码已公开于 https://github.com/Rainier-rq/verl-if。
原文摘要 · Abstract (English)
Reasoning models excel in complex problem solving but exhibit a concerning trade off between reasoning capabilities and instruction following abilities. Existing approaches for improving instruction following rely on stronger external models, creating methodological bottlenecks and practical limitations including increased costs and accessibility constraints. We propose a self-supervised RL framework that leverages reasoning models' own internal signals to improve instruction following capabilities without external supervision. Extensive experiments demonstrate that our framework significantly improves instruction following capabilities while maintaining reasoning performance, offering a scalable and cost-effective approach to enhance instruction following in reasoning models. The data and code are publicly available at https://github.com/Rainier-rq/verl-if.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。