用强化学习动态组合多个模型和技能,提升智能体在复杂任务中的表现。
Maestro: Reinforcement Learning to Orchestrate Hierarchical Model-Skill Ensembles

- 通过强化学习动态调度专家模型与双层技能库,实现灵活协作。
- 仅用40亿参数的调度器,在10个基准上平均准确率达70.1%,超越GPT-5和Gemini。
- 可泛化到未见模型与技能,无需重训,适合多模态任务部署。
大型语言模型(LLMs)与模块化技能的兴起赋予自主代理更强能力。现有框架通常依赖单一模型和固定逻辑调用技能,导致无法充分利用不同模型在各领域的互补优势,限制下游任务性能。本文提出Maestro(Multimodal Agent for Expert-Skill Targeted Reinforced Orchestration),一个基于强化学习的编排框架,将异构多模态任务重构为对分层模型-技能注册表的序列决策过程。不将知识集中于单一模型,而是训练轻量级策略动态组合冻结的专家模型与两级技能库,每一步决定是否调用外部专家、选择哪对模型-技能,以及何时终止。策略通过结果导向的强化学习优化,无需步骤级标注。在涵盖数学推理、图表理解、高分辨率感知和领域特定分析的十个代表性多模态基准上评估,仅使用40亿参数的编排器,平均准确率达70.1%,超过GPT-5(69.3%)和Gemini-2.5-Pro(68.7%)。关键的是,学习到的协调策略可泛化至未见模型与技能,无需重训:在四个挑战性基准上,引入域外专家后平均达59.5%,优于所有闭源基线。Maestro还保持高计算效率,延迟低。源代码已公开。
原文摘要 · Abstract (English)
The proliferation of large language models (LLMs) and modular skills has endowed autonomous agents with increasingly powerful capabilities. Existing frameworks typically rely on monolithic LLMs and fixed logic to interface with these skills. This gives rise to a critical bottleneck: different LLMs offer distinct advantages across diverse domains, yet current frameworks fail to exploit the complementary strengths of models and skills, thereby limiting their performance on downstream tasks. In this paper, we present Maestro (Multimodal Agent for Expert-Skill Targeted Reinforced Orchestration), a Reinforcement Learning (RL)-driven orchestration framework that reframes heterogeneous multimodal tasks as a sequential decision-making process over a hierarchical model-skill registry. Rather than consolidating all knowledge into a single model, Maestro trains a lightweight policy to dynamically compose ensembles of frozen expert models and a two-tier skill library, deciding at each step whether to invoke an external expert, which model-skill pair to select, and when to terminate. The policy is optimized via outcome-based RL, requiring no step-level supervision. We evaluate Maestro across ten representative multimodal benchmarks spanning mathematical reasoning, chart understanding, high-resolution perception, and domain-specific analysis. With only a 4B orchestrator, Maestro achieves an average accuracy of 70.1%, surpassing both GPT-5 (69.3%) and Gemini-2.5-Pro (68.7%). Crucially, the learned coordination policy generalizes to unseen models and skills without retraining: augmenting the registry with out-of-domain experts yields a 59.5% average on four challenging benchmarks, outperforming all closed-source baselines. Maestro further maintains high computational efficiency with low latency. The source code is available at https://github.com/jinyangwu/Maestro.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。