通过重构难题让大模型先学简单版本,再提升原问题解题能力。
Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problems
- 将开放问题转为选择题等简化形式,降低搜索空间。
- 在6个基准上平均提升4.72%(Qwen)和3.23%(Llama)。
- 适合想提升模型推理能力的研究者与工程师。
基于可验证奖励的强化学习(RLVR)提升了大模型的推理能力,但存在根本局限:当前策略无法解决过于困难的问题,导致无有效奖励信号。本文提出一种基于任务重构的简单而有效方案,将复杂的开放性问题转化为认知上更简单的变体(如多项选择、填空题),保留原始答案的同时缩小有效搜索空间,提供更密集的学习信号。这些重构形式构成从判别到生成的连续谱,用于构建自适应课程:模型先从结构化、较易格式学习,知识再反向迁移至原始开放问题。基于此,我们提出Cog-DRIFT框架,动态生成重构实例并按难度组织课程。训练从易到难推进,使模型能从以往无信号的难题中学习。实验表明,该方法在原本无法求解的难题上取得绝对提升(Qwen +10.11%,Llama +8.64%),且在其他未见数据集上具有良好泛化性。跨2个模型与6个推理基准,平均优于标准GRPO与强基线+4.72%(Qwen)和+3.23%(Llama)。此外,测试时pass@k得分提升,课程设计也提高样本效率。结果表明,任务重构与课程学习是突破大模型后训练探索障碍的有效范式。
原文摘要 · Abstract (English)
Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of LLMs, yet a fundamental limitation remains: models cannot learn from problems that are too difficult to solve under their current policy, as these yield no meaningful reward signal. We propose a simple yet effective solution based on task reformulation. We transform challenging open-ended problems into cognitively simpler variants -- such as multiple-choice and cloze formats -- that preserve the original answer while reducing the effective search space and providing denser learning signals. These reformulations span a spectrum from discriminative to generative tasks, which we exploit to bootstrap learning: models first learn from structured, easier formats, and this knowledge transfers back to improve performance on the original open-ended problems. Building on this insight, we introduce Cog-DRIFT, a framework that constructs reformulated variants and organizes them into an adaptive curriculum based on difficulty. Training progresses from easier to harder formats, enabling the model to learn from problems that previously yielded zero signal under standard RL post-training. Cog-DRIFT not only improves on the originally unsolvable hard problems (absolute +10.11% for Qwen and +8.64% for Llama) but also generalizes well to other held-out datasets. Across 2 models and 6 reasoning benchmarks, our method consistently outperforms standard GRPO and strong guided-exploration baselines. On average, Cog-DRIFT shows +4.72% (Qwen) and +3.23% (Llama) improvements over the second-best baseline. We further show that Cog-DRIFT improves pass@k at test time, and the curriculum improves sample efficiency. Overall, our results highlight task reformulation and curriculum learning as an effective paradigm for overcoming the exploration barrier in LLM post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。