让智能体在动态环境里实时组合世界模型,提升适应能力。
Test-Time Mixture of World Models for Embodied Agents in Dynamic Environments
- 测试时动态调整模型组合方式,实现持续自适应。
- 零样本和少样本场景下均表现优异,超越传统方法。
- 适合需要快速应对新环境的机器人与虚拟代理应用。
基于语言模型的具身智能体正越来越多地部署于真实场景中。然而,在动态环境中,其适应能力仍受限,准确且灵活的世界模型构建对有效推理与决策至关重要。为解决此问题,本文将专家混合(Mixture-of-Experts, MoE)范式扩展至具身智能体。传统MoE虽将知识模块化并预训练路由机制,但部署后无法更新,难以适应未见领域。为此,我们提出测试时世界模型混合(Test-time Mixture of World Models, TMoW),可在测试阶段动态更新路由函数,使智能体能重组现有模型并融合新模型以实现持续适应。该方法包含三部分:(i) 多粒度原型路由,根据对象到场景级别的相似性动态调整混合;(ii) 测试时微调,将未见域特征与原型对齐;(iii) 由少量数据和已有原型高效生成新模型。在VirtualHome、ALFWorld和RLBench基准上评估显示,TMoW在零样本适应与少样本扩展场景中均表现强劲,证明其可有效支持具身智能体在动态环境中的运行。
原文摘要 · Abstract (English)
Language model (LM)-based embodied agents are increasingly deployed in real-world settings. Yet, their adaptability remains limited in dynamic environments, where constructing accurate and flexible world models is crucial for effective reasoning and decision-making. To address this challenge, we extend the Mixture-of-Experts (MoE) paradigm to embodied agents. While conventional MoE architectures modularize knowledge into expert components with pre-trained routing, they remain rigid once deployed, making them less effective for adapting to unseen domains in dynamic environments. We therefore propose Test-time Mixture of World Models (TMoW), a framework that enhances adaptability to unseen and evolving domains. TMoW updates its routing function over world models at test time, unlike conventional MoE where the function remains fixed, enabling agents to recombine existing models and integrate new ones for continual adaptation. It achieves this through (i) multi-granular prototype-based routing, which adapts mixtures across object- to scene-level similarities, (ii) test-time refinement that aligns unseen domain features with prototypes during inference, and (iii) distilled mixture-based augmentation, which efficiently constructs new models from few-shot data and existing prototypes. We evaluate TMoW on VirtualHome, ALFWorld, and RLBench benchmarks, demonstrating strong performance in both zero-shot adaptation and few-shot expansion scenarios, and showing that it enables embodied agents to operate effectively in dynamic environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。