让多模态智能体协同工作,解决复杂任务中的跨模态协作难题。
Orchestra-o1: Omnimodal Agent Orchestration

- 设计统一调度机制,支持文本、图像、音频、视频等多模态任务分解与并行执行。
- 在OmniGAIA基准上比次优方法高10.3%准确率,实现当前开源多模态智能体最佳性能。
- 采用新型强化学习训练方法,适合需要多模态理解与协作的现实场景应用。
近年来,智能体集群的成功推动了大语言模型驱动智能体从单体流程转向多智能体系统,凸显了智能体编排在任务分解与协作中的重要性。然而,现有编排框架仅限于有限模态,难以泛化到异构模态共存交互的复杂场景。这一局限在多模态情境中尤为突出,任务需统一理解与协调文本、图像、音频、视频等多种输入。本文提出Orchestra-o1,一种支持多模态智能体高效协作的编排框架。该框架引入统一编排机制,实现模态感知的任务分解、在线子智能体专业化和并行子任务执行。其可扩展设计使系统能有效应对涉及异构信息源的复杂现实任务,在OmniGAIA基准上超越次优方法10.3%的准确率。此外,我们提出决策对齐组相对策略优化(DA-GRPO),一种高效的智能体强化学习方法,用于训练Orchestra-o1-8B,该模型在所有现有开源多模态智能体中达到最先进水平。
原文摘要 · Abstract (English)
The recent success of agent swarms has shifted the paradigm of large language model (LLM)-based agents from single-agent workflows to multi-agent systems, highlighting the importance of agent orchestration for task decomposition and collaboration. However, existing orchestration frameworks are limited to a narrow set of modalities and struggle to generalize to more complex settings where heterogeneous modalities coexist and interact. This limitation becomes particularly pronounced in omnimodal scenarios, where tasks require the unified understanding and coordination of diverse inputs such as text, image, audio, and video. In this work, we propose Orchestra-o1, an omnimodal agent orchestration framework designed to support efficient agent collaboration across multiple modalities. Orchestra-o1 introduces a unified orchestration mechanism that enables modality-aware task decomposition, online sub-agent specialization, and parallel sub-task execution. This scalable design allows agent systems to effectively tackle complex real-world tasks involving heterogeneous information sources, surpassing the second-best approach by 10.3% accuracy on the OmniGAIA benchmark. Furthermore, we introduce decision-aligned group relative policy optimization (DA-GRPO), an efficient agentic reinforcement learning approach for training Orchestra-o1-8B, which also achieves state-of-the-art performance against all existing open-source omnimodal agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。