通过迭代优化提升大模型的递归推理能力，仅用3000样本提升20%表现
Unlocking Recursive Thinking of LLMs: Alignment via Refinement
- 设计可微分的修正机制，融合批评与改进动作
- 仅用3000条合成数据，使LLaMA-3-8B在AlpacaEval上胜率提升超20%
- 适合需要高效提升推理能力的模型部署场景
OpenAI o1系列模型表明，使用长链思维（CoT）可显著提升性能。然而，大语言模型的递归思考能力仍受限，尤其缺乏专家标注数据用于蒸馏。本文提出一种新方法——基于修正的对齐（AvR），旨在通过长链思维释放大模型的递归推理潜力。AvR引入一个整合批评与改进动作的修正过程，借助可微学习技术优化修正感知奖励。由此生成的多轮数据可组织为长修正思维链，支持测试时扩展。实验表明，该方法显著优于传统偏好优化。值得注意的是，仅使用3000条合成样本，即在AlpacaEval 2.0上使LLaMA-3-8B-Instruct模型胜率提升超过20%。代码已开源。
原文摘要 · Abstract (English)
The OpenAI o1-series models have demonstrated that leveraging long-form Chain of Thought (CoT) can substantially enhance performance. However, the recursive thinking capabilities of Large Language Models (LLMs) remain limited, particularly in the absence of expert-curated data for distillation. In this paper, we propose \textbf{AvR}: \textbf{Alignment via Refinement}, a novel method aimed at unlocking the potential of LLMs for recursive reasoning through long-form CoT. AvR introduces a refinement process that integrates criticism and improvement actions, guided by differentiable learning techniques to optimize \textbf{refinement-aware rewards}. As a result, the synthesized multi-round data can be organized as a long refinement thought, further enabling test-time scaling. Experimental results show that AvR significantly outperforms conventional preference optimization methods. Notably, with only 3k synthetic samples, our method boosts the performance of the LLaMA-3-8B-Instruct model by over 20\% in win rate on AlpacaEval 2.0. Our code is available at Github (https://github.com/Banner-Z/AvR.git).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。