AlphaApollo通过多轮推理与进化提升大模型复杂问题求解能力。
AlphaApollo: A System for Deep Agentic Reasoning
- 设计三组件系统：多轮代理推理、强化学习优化工具使用、验证驱动的迭代演化
- 在数学推理基准上实现超85%工具调用成功率，性能提升最高达20.35%
- 适合研究长程推理、智能体系统或需要可靠工具协同的开发者
我们提出AlphaApollo，一个针对基础模型推理中两大瓶颈的智能体推理系统：（1）复杂长程问题求解时推理能力有限；（2）测试时演进不可靠，缺乏可信验证。AlphaApollo通过三个组件协调模型与工具：（i）多轮代理推理，以结构化工具调用与响应形式化模型-环境交互；（ii）多轮代理学习，采用逐轮强化学习优化工具使用推理，同时解耦动作与工具响应以实现稳定训练；（iii）多轮代理演化，通过提出-评判-更新循环结合工具辅助验证与长程记忆不断精炼解决方案。在七个数学推理基准和多个模型规模下，AlphaApollo展现出可靠工具使用（>85%工具调用成功率）、显著的多轮强化学习增益（例如，Qwen2.5-1.5B-Instruct从1.07%提升至9.64%，Qwen2.5-7B-Instruct从8.77%提升至20.35%），以及演化带来的改进（如，Qwen2.5-3B-Instruct从5.27%提升至7.70%，Qwen2.5-14B-Instruct从16.53%提升至21.08%）。该项目仍在持续进行中，欢迎社区反馈，将持续更新源码与技术报告。
原文摘要 · Abstract (English)
We present AlphaApollo, an agentic reasoning system that targets two bottlenecks in foundation-model reasoning: (1) limited reasoning capacity for complex, long-horizon problem solving and (2) unreliable test-time evolution without trustworthy verification. AlphaApollo orchestrates models and tools via three components: (i) multi-turn agentic reasoning, which formalizes model-environment interaction with structured tool calls and responses; (ii) multi-turn agentic learning, which applies turn-level reinforcement learning to optimize tool-use reasoning while decoupling actions from tool responses for stable training; and (iii) multi-round agentic evolution, which refines solutions through a propose-judge-update loop with tool-assisted verifications and long-horizon memory. Across seven math reasoning benchmarks and multiple model scales, AlphaApollo improves performance through reliable tool use (> 85% tool-call success), substantial gains from multi-turn RL (Avg@32: Qwen2.5-1.5B-Instruct 1.07% -> 9.64%, Qwen2.5-7B-Instruct 8.77% -> 20.35%), and improvements from evolution (e.g., Qwen2.5-3B-Instruct 5.27% -> 7.70%, Qwen2.5-14B-Instruct 16.53% -> 21.08%). This project is still ongoing. We welcome feedback from the community and will frequently update the source code and technical report.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。