让大模型学会深度思考复杂指令，效果接近更大模型。
Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models
- 用可验证奖励强化学习，引导模型深入分析指令结构。
- 1.5B模型在复杂指令上提升11.74%，媲美8B模型。
- 适合需要精准理解多条件指令的AI应用开发者。
现有大语言模型在应对复杂指令时表现不佳，尤其当多个约束以并行、链式或分支结构共存时。尽管思维链（CoT）被寄予厚望，但原始CoT因仅简单复述指令而产生负面效果，无法揭示约束间的层级与维度关系。为此，本文提出RAIF方法，通过激励推理实现测试阶段计算扩展，以增强模型处理复杂指令的能力。首先，基于现有分类体系分解复杂指令，并提出可复现的数据采集方法；其次，采用可验证规则中心奖励信号的强化学习，结合样本级对比策略，强化高质量思维链生成；同时利用专家行为克隆，推动模型从快速推理向高阶推理平稳迁移。在七个综合基准上的实证表明，1.5B模型性能提升11.74%，达到与8B模型相当水平；在分布外约束下也展现良好泛化能力。代码与数据已公开于https://github.com/yuleiqin/RAIF。
原文摘要 · Abstract (English)
Existing large language models (LLMs) face challenges of following complex instructions, especially when multiple constraints are present and organized in paralleling, chaining, and branching structures. One intuitive solution, namely chain-of-thought (CoT), is expected to universally improve capabilities of LLMs. However, we find that the vanilla CoT exerts a negative impact on performance due to its superficial reasoning pattern of simply paraphrasing the instructions. It fails to peel back the compositions of constraints for identifying their relationship across hierarchies of types and dimensions. To this end, we propose RAIF, a systematic method to boost LLMs in dealing with complex instructions via incentivizing reasoning for test-time compute scaling. First, we stem from the decomposition of complex instructions under existing taxonomies and propose a reproducible data acquisition method. Second, we exploit reinforcement learning (RL) with verifiable rule-centric reward signals to cultivate reasoning specifically for instruction following. We address the shallow, non-essential nature of reasoning under complex instructions via sample-wise contrast for superior CoT enforcement. We also exploit behavior cloning of experts to facilitate steady distribution shift from fast-thinking LLMs to skillful reasoners. Extensive evaluations on seven comprehensive benchmarks confirm the validity of the proposed method, where a 1.5B LLM achieves 11.74% gains with performance comparable to a 8B LLM. Evaluation on OOD constraints also confirms the generalizability of our RAIF. Codes and data are available at https://github.com/yuleiqin/RAIF. Keywords: reinforcement learning with verifiable rewards (RLVR), instruction following, complex instructions
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。