通过对比成功与失败路径,高效合成高质量推理数据。
Learning from Contrasts: Synthesizing Reasoning Paths from Diverse Search Trajectories

- 利用高/低质量路径差异,提炼关键策略与错误模式
- 仅用6万条合成数据达到590万条原始数据效果
- 适合需要强泛化能力的推理模型训练
蒙特卡洛树搜索(MCTS)广泛用于自动化推理数据探索,但现有监督提取方法效率低下。标准方法仅保留最高奖励轨迹,丢弃了众多探索路径中的对比信号。本文提出对比推理路径合成(CRPS)框架,将监督提取从过滤过程转变为合成过程。CRPS通过结构化反思机制分析高质与低质搜索轨迹的差异,提取关于策略转折点和局部失败模式的显式信息,指导生成融合成功模式并规避已知陷阱的推理链。实验表明,仅用6万条CRPS合成数据微调的模型,在性能上可媲美或超越使用590万条标准拒绝采样数据训练的基线模型,实现20倍的数据量缩减。此外,CRPS在跨域基准测试中提升泛化能力,证明从成功与失败的对比中学习,能产生比仅学习成功更可迁移的推理能力。
原文摘要 · Abstract (English)
Monte Carlo Tree Search (MCTS) has been widely used for automated reasoning data exploration, but current supervision extraction methods remain inefficient. Standard approaches retain only the single highest-reward trajectory, discarding the comparative signals present in the many explored paths. Here we introduce \textbf{Contrastive Reasoning Path Synthesis (CRPS)}, a framework that transforms supervision extraction from a filtering process into a synthesis procedure. CRPS uses a structured reflective process to analyze the differences between high- and low-quality search trajectories, extracting explicit information about strategic pivots and local failure modes. These insights guide the synthesis of reasoning chains that incorporate success patterns while avoiding identified pitfalls. We show empirically that models fine-tuned on just 60K CRPS-synthesized examples match or exceed the performance of baselines trained on 590K examples derived from standard rejection sampling, a 20$\times$ reduction in dataset size. Furthermore, CRPS improves generalization on out-of-domain benchmarks, demonstrating that learning from the contrast between success and failure produces more transferable reasoning capabilities than learning from success alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。