让大模型学会预览与自检,提升复杂指令遵循能力
Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following
- 通过预览和自检机制,强制模型严谨推理
- 小样本高质量数据集训练,32B模型超越更大模型
- 适合需要精准执行复杂指令的场景
尽管大语言模型在数学、编程和通用谜题上的推理能力显著提升,但在遵循复杂指令时仍表现不一。研究发现,思维阶段的懒惰推理是导致指令遵循不佳的主要原因。为此,提出一个包含预览与自检的完整框架:首先生成带复杂约束的指令并过滤,构建难、易、通过三类提示数据集;再对通过类提示使用拒绝采样,建立小而高质量的数据集,实现模型冷启动初始化;随后采用保持熵的监督微调(Entropy-SFT)结合基于规则密集奖励的词级熵自适应强化学习(TEA-RL),引导模型转变推理机制,形成具备预览与自检能力的泛化推理模式。在多个指令遵循基准测试中,不同规模模型均显著提升,其中Light-IF-32B模型性能超越更大的开源模型DeepSeek-R1及闭源模型Doubao-1.6。
原文摘要 · Abstract (English)
While advancements in the reasoning abilities of LLMs have significantly enhanced their performance in solving mathematical problems, coding tasks, and general puzzles, their effectiveness in accurately adhering to instructions remains inconsistent, particularly with more complex directives. Our investigation identifies lazy reasoning during the thinking stage as the primary factor contributing to poor instruction adherence. To mitigate this issue, we propose a comprehensive framework designed to enable rigorous reasoning processes involving preview and self-checking, essential for satisfying strict instruction constraints. Specifically, we first generate instructions with complex constraints and apply a filtering process to obtain valid prompts, resulting in three distinct prompt datasets categorized as hard, easy, and pass. Then, we employ rejection sampling on the pass prompts to curate a small yet high-quality dataset, enabling a cold-start initialization of the model and facilitating its adaptation to effective reasoning patterns. Subsequently, we employ an entropy-preserving supervised fine-tuning (Entropy-SFT) strategy coupled with token-wise entropy-adaptive (TEA-RL) reinforcement learning guided by rule-based dense rewards. This approach encourages the model to transform its reasoning mechanism, ultimately fostering generalizable reasoning abilities that encompass preview and self-checking. Extensive experiments conducted on instruction-following benchmarks demonstrate remarkable performance improvements across various model scales. Notably, our Light-IF-32B model surpasses both larger open-source models such as DeepSeek-R1 and closed-source models like Doubao-1.6.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。