通过自追踪推理步骤优化，让大模型更精准简洁地思考。
SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression
- 用模型自身生成的推理偏好信号，逐步优化思考过程。
- 推理更准确且更简短，避免冗余思考，性能不下降。
- 无需额外模型或人工标注，适合多语言和多种任务。
测试时扩展已被证明能进一步提升预训练大模型的性能。然而，主流的后训练方法（如基于思维链的强化学习）常因需要辅助模型和过度思考而带来巨大计算开销。本文实证发现，错误答案部分源于缺乏正确自我修正的冗长推理过程，错误在多个推理步骤中累积。为此，我们提出自追踪分步偏好优化（SSPO），一种可插拔的强化学习过程监督框架，实现对每个推理步骤的细粒度优化。SSPO无需辅助模型，也无需逐步人工标注，而是利用模型自身生成的分步偏好信号来指导推理压缩优化。实验表明，SSPO生成的推理序列既准确又简洁，有效缓解了过度思考行为，且在多种领域和语言下均保持模型性能。
原文摘要 · Abstract (English)
Test-time scaling has proven effective in further enhancing the performance of pretrained Large Language Models (LLMs). However, mainstream post-training methods (i.e., reinforcement learning (RL) with chain-of-thought (CoT) reasoning) often incur substantial computational overhead due to auxiliary models and overthinking. In this paper, we empirically reveal that the incorrect answers partially stem from verbose reasoning processes lacking correct self-fix, where errors accumulate across multiple reasoning steps. To this end, we propose Self-traced Step-wise Preference Optimization (SSPO), a pluggable RL process supervision framework that enables fine-grained optimization of each reasoning step. Specifically, SSPO requires neither auxiliary models nor stepwise manual annotations. Instead, it leverages step-wise preference signals generated by the model itself to guide the optimization process for reasoning compression. Experiments demonstrate that the generated reasoning sequences from SSPO are both accurate and succinct, effectively mitigating overthinking behaviors without compromising model performance across diverse domains and languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。