逆向挖掘推理过程,让模型学会创造性写作。
Reverse-Engineered Reasoning for Open-Ended Generation
- 从优质结果反推推理路径,无需奖励信号或昂贵蒸馏。
- 构建2万条开放生成推理轨迹,训练出80亿参数模型。
- 性能媲美甚至超越GPT-4o等闭源大模型,适合创意生成任务。
尽管‘深度推理’范式在数学等可验证领域取得显著进展,但在开放性、创造性生成任务中仍面临重大挑战。当前主流方法——强化学习(RL)和指令蒸馏——在此领域表现不佳:RL缺乏明确奖励信号与高质量奖励模型,而蒸馏成本过高且受限于教师模型能力。为此,我们提出逆向工程推理(REER),从根本上改变思路:不通过试错或模仿正向构建推理过程,而是从已知优质结果反向计算可能生成这些结果的潜在线索式推理路径。采用这种可扩展、无梯度的方法,我们构建并开源了包含20,000条深度推理轨迹的DeepWriting-20K数据集。基于该数据训练的DeepWriter-8B模型,不仅超越多个开源基线,还达到与领先闭源模型如GPT-4o和Claude 3.5相当甚至更优的性能。
原文摘要 · Abstract (English)
While the ``deep reasoning'' paradigm has spurred significant advances in verifiable domains like mathematics, its application to open-ended, creative generation remains a critical challenge. The two dominant methods for instilling reasoning -- reinforcement learning (RL) and instruction distillation -- falter in this area; RL struggles with the absence of clear reward signals and high-quality reward models, while distillation is prohibitively expensive and capped by the teacher model's capabilities. To overcome these limitations, we introduce REverse-Engineered Reasoning (REER), a new paradigm that fundamentally shifts the approach. Instead of building a reasoning process ``forwards'' through trial-and-error or imitation, REER works ``backwards'' from known-good solutions to computationally discover the latent, step-by-step deep reasoning process that could have produced them. Using this scalable, gradient-free approach, we curate and open-source DeepWriting-20K, a large-scale dataset of 20,000 deep reasoning trajectories for open-ended tasks. Our model, DeepWriter-8B, trained on this data, not only surpasses strong open-source baselines but also achieves performance competitive with, and at times superior to, leading proprietary models like GPT-4o and Claude 3.5.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。