用在线多智能体协作生成更优数学推理链，提升大模型解题能力。
Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning
- 多个小模型分步协作生成推理链，实时更新策略。
- 相比直接推理，推理链质量显著提升，任务准确率更高。
- 适合需要精细推理的数学题生成与模型优化场景。
数学推理是大语言模型的重要能力，但生成详尽准确的推理过程仍具挑战。本文提出一种基于在线学习的新型方法——流式协作（Flow-DPO），通过多个组件模型在迭代中协同构建解题路径。该方法采用在线直接偏好优化（DPO）进行训练，为每个样本生成偏好对并实时更新模型。实验对比了本方法与直接推理生成的推理链质量，证明其在数学推理任务中能有效提升模型表现。
原文摘要 · Abstract (English)
Mathematical reasoning is a crucial capability for Large Language Models (LLMs), yet generating detailed and accurate reasoning traces remains a significant challenge. This paper introduces a novel approach to produce high-quality reasoning traces for LLM fine-tuning using online learning \textbf{Flows}. Our method employs an incremental output production Flow, where component LLMs collaboratively construct solutions through iterative communication. We train the Flow using online Direct Preference Optimization (DPO) learning with rollouts, generating DPO pairs for each training example and updating models in real-time. We directly compare the quality of reasoning traces generated by our method with those produced through direct model inference, demonstrating the effectiveness of our approach in improving LLM performance in mathematical reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。