让AI模型多试几条思路再选最优解，提升推理准确性和速度
DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs
- 每个智能体生成多个草稿，通过同伴评估和奖励模型筛选最佳路径
- 在代码生成、数学推理等任务上准确率和收敛速度显著优于现有方法
- 适合需要高可靠推理的复杂任务，如编程与知识密集型问答
大型语言模型在多步推理和问题解决中展现出强大能力。近期研究引入多智能体反思框架，利用强化学习让多个模型相互批判和优化输出。然而，这些方法通常依赖单次生成，缺乏推理探索的结构多样性。本文提出DRAFT-RL，将链式草稿（Chain-of-Draft, CoD）推理融入多智能体强化学习训练中。每个智能体对每个查询生成多个草稿，由同伴智能体和学习到的奖励模型评估，选出最具潜力的推理轨迹，并通过演员-评论家学习优化后续推理策略。DRAFT-RL实现了显式的多路径探索、同伴引导的反思与奖励对齐的选择，使大模型智能体行为更稳健且可解释。我们在代码生成、符号数学和知识密集型问答等复杂推理任务上进行评估，结果表明DRAFT-RL在准确率和收敛速度上均显著优于现有的反思型与强化学习基线模型。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown impressive capabilities in multi-step reasoning and problem-solving.Recent works introduce multi-agent reflection frameworks where multiple LLM agents critique and refine each other's outputs using reinforcement learning (RL). However, these approaches often rely on single-shot responses and lack structural diversity in reasoning exploration. In this paper, we propose DRAFT-RL, a novel framework that integrates Chain-of-Draft (CoD) reasoning into multi-agent RL training. Instead of generating single responses, each agent produces multiple drafts per query, which are then evaluated by peer agents and a learned reward model to identify the most promising trajectory. These selected drafts are used to refine future reasoning strategies through actor-critic learning.DRAFT-RL enables explicit multi-path exploration, peer-guided reflection, and reward-aligned selection, resulting in more robust and interpretable LLM agent behavior. We evaluate our method on complex reasoning tasks including code synthesis, symbolic math, and knowledge-intensive QA,demonstrating that DRAFT-RL outperforms existing reflective and RL-based agents by significant margins in both accuracy and convergence speed
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。