让AI模型通过自我辩论实现推理能力持续进化
DEBATE, TRAIN, EVOLVE: Self Evolution of Language Model Reasoning
- 多智能体辩论生成训练信号，无需人工标注
- 在GSM-PLUS上平均准确率提升8.92%
- 跨领域泛化能力强，适合追求自进化AI的研究者
大型语言模型通过海量数据训练已显著提升推理能力，但单纯依赖更多数据已不现实，亟需模型自主改进推理能力。本文提出无真值训练框架DTE，利用多智能体辩论痕迹演化单个语言模型。引入新提示策略Reflect-Critique-Refine，显式指导智能体批判与优化推理过程。在七个推理基准上对六种开源模型的评估显示，该框架在挑战性GSM-PLUS数据集上平均准确率提升8.92%；在其余所有基准上平均提升5.8%，表现出强跨领域泛化能力，说明方法捕获了通用推理能力。代码与训练模型已在GitHub公开。
原文摘要 · Abstract (English)
Large language models (LLMs) have improved significantly in their reasoning through extensive training on massive datasets. However, relying solely on additional data for improvement is becoming increasingly impractical, highlighting the need for models to autonomously enhance their reasoning without external supervision. In this paper, we propose Debate, Train, Evolve (DTE), a novel ground truth-free training framework that uses multi-agent debate traces to evolve a single language model. We also introduce a new prompting strategy Reflect-Critique-Refine, to improve debate quality by explicitly instructing agents to critique and refine their reasoning. Extensive evaluations on seven reasoning benchmarks with six open-weight models show that our DTE framework achieve substantial improvements, with an average accuracy gain of 8.92% on the challenging GSM-PLUS dataset. Furthermore, we observe strong cross-domain generalization, with an average accuracy gain of 5.8% on all other benchmarks, suggesting that our method captures general reasoning capabilities. Our framework code and trained models are publicly available at https://github.com/ctrl-gaurav/Debate-Train-Evolve
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。