arXiv:2509.02322cs.CV2025-09被引 5

提出通用智能体OmniActor,能无缝切换2D与3D环境执行任务。

OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds

  • 通过分层异构MoE架构分离深层参数,解决GUI与具身数据冲突。
  • 在GUI和具身任务上均超越单一环境训练的模型,提升显著。
  • 统一动作空间并融合多源数据,适合复杂跨环境任务场景。

多模态大语言模型正向具备主动执行任务能力的多模态智能体演进。现有研究多聚焦于GUI或具身场景,分别对应与2D虚拟世界或3D真实世界的交互。然而,许多复杂任务需在两类环境中交替操作。我们初期混合训练时发现数据冲突导致性能下降。深入分析表明,GUI与具身数据在浅层具有协同性,在深层存在冲突,类似人脑皮层与小脑的机制。为此,我们提出高性能通用智能体OmniActor,从结构与数据双重角度设计。首先,提出层异构MoE,通过分离深层参数消除冲突,共享浅层参数利用协同效应。由此实现优于仅在单一环境训练的模型。此外,统一GUI与具身任务的动作空间,并整合多源大规模数据进行训练,显著提升跨场景表现,尤其在GUI任务中效果突出。代码将公开。

原文摘要 · Abstract (English)

Multimodal large language models are evolving toward multimodal agents capable of proactively executing tasks. Most agent research focuses on GUI or embodied scenarios, which correspond to agents interacting with 2D virtual worlds or 3D real worlds, respectively. However, many complex tasks typically require agents to interleavely interact with these two types of environment. We initially mix GUI and embodied data to train, but find the performance degeneration brought by the data conflict. Further analysis reveals that GUI and embodied data exhibit synergy and conflict at the shallow and deep layers, respectively, which resembles the cerebrum-cerebellum mechanism in the human brain. To this end, we propose a high-performance generalist agent OmniActor, designed from both structural and data perspectives. First, we propose Layer-heterogeneity MoE to eliminate the conflict between GUI and embodied data by separating deep-layer parameters, while leverage their synergy by sharing shallow-layer parameters. By successfully leveraging the synergy and eliminating the conflict, OmniActor outperforms agents only trained by GUI or embodied data in GUI or embodied tasks. Furthermore, we unify the action spaces of GUI and embodied tasks, and collect large-scale GUI and embodied data from various sources for training. This significantly improves OmniActor under different scenarios, especially in GUI tasks. The code will be publicly available.

通用智能体多模态跨环境任务执行

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。