用多智能体协作提升大模型数学推理准确率
Mars-PO: Multi-Agent Reasoning System Preference Optimization
- 构建多智能体系统，融合优质输出生成正样本对
- 在MATH数据集上将推理准确率从50.38%提升至57.82%
- 适合需要高精度数学推理的AI系统开发者
数学推理是大语言模型的核心能力，但自回归生成过程易导致错误、幻觉和不一致，尤其在多步推理中。本文提出Mars-PO框架，通过多智能体系统整合多个智能体的高质量输出，形成混合正样本集，并与各智能体专属的负样本配对，构建鲁棒的偏好训练对。该方法在共享正样本对齐智能体的同时，针对性弥补个体缺陷，在数学推理基准测试中表现显著提升。例如，对最先进的指令微调模型Llama3.1-8B-Instruct，其在MATH基准上的准确率从50.38%提升至57.82%。实验结果进一步表明，该方法优于监督微调、基础DPO及其改进版本，验证了其有效性。
原文摘要 · Abstract (English)
Mathematical reasoning is a fundamental capability for large language models (LLMs), yet achieving high performance in this domain remains a significant challenge. The auto-regressive generation process often makes LLMs susceptible to errors, hallucinations, and inconsistencies, particularly during multi-step reasoning. In this paper, we propose Mars-PO, a novel framework to improve the mathematical reasoning capabilities of LLMs through a multi-agent system. It combines high-quality outputs from multiple agents into a hybrid positive sample set and pairs them with agent-specific negative samples to construct robust preference pairs for training. By aligning agents with shared positive samples while addressing individual weaknesses, Mars-PO achieves substantial performance improvements on mathematical reasoning benchmarks. For example, it increases the accuracy on the MATH benchmark of the state-of-the-art instruction-tuned LLM, Llama3.1-8B-Instruct, from 50.38% to 57.82%. Experimental results further demonstrate that our method consistently outperforms other baselines, such as supervised fine-tuning, vanilla DPO, and its enhanced versions, highlighting the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。