反向数据提升推理性能，混合使用反而降低效果。
When Inverse Data Outperforms: Exploring the Pitfalls of Mixed Data in Multi-Stage Fine-Tuning
- 用反向推理数据训练，比正向数据更优。
- 混合正反数据后，准确率下降1.6%至6.8%。
- 适合研究方向敏感型对齐方法的学者参考。
现有研究表明，少量数据蒸馏即可实现o1级性能，但多数方法仅关注单向监督微调（SFT），忽视了不同推理模式间的复杂交互。本文构建了r1k数据集，通过反转s1k中1000个正向示例生成高质量反向推理数据，并考察SFT与直接偏好优化（DPO）在双向推理目标下的对齐效果。在多个基准测试中，基于r1k的SFT相比s1k提升1.6%至6.8%的准确率。然而，将正向与反向数据混合进行SFT会削弱方向区分性。尽管DPO可部分恢复方向性，但会抑制非偏好推理路径，导致概率质量向无关输出偏移。结果表明，混合推理数据引入冲突监督信号，亟需具备方向感知能力的鲁棒对齐策略。
原文摘要 · Abstract (English)
Existing work has shown that o1-level performance can be achieved with limited data distillation, but most existing methods focus on unidirectional supervised fine-tuning (SFT), overlooking the intricate interplay between diverse reasoning patterns. In this paper, we construct r1k, a high-quality reverse reasoning dataset derived by inverting 1,000 forward examples from s1k, and examine how SFT and Direct Preference Optimization (DPO) affect alignment under bidirectional reasoning objectives. SFT on r1k yields a 1.6%--6.8% accuracy improvement over s1k across evaluated benchmarks. However, naively mixing forward and reverse data during SFT weakens the directional distinction. Although DPO can partially recover this distinction, it also suppresses less preferred reasoning paths by shifting the probability mass toward irrelevant outputs. These findings suggest that mixed reasoning data introduce conflicting supervision signals, underscoring the need for robust and direction-aware alignment strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。