arXiv:2506.24019cs.CVcs.CL2025-06被引 3

Ella是能长期学习的虚拟社交机器人,靠记忆与人互动进化。

Ella: Embodied Social Agents with Lifelong Memory

  • 用语义+时空双记忆系统存取知识和经历
  • 15个智能体共存数日,能领导协作达成目标
  • 适合研究具身智能与社会交互的学者

我们提出Ella,一个在3D开放世界中具备持续学习能力的具身社交智能体,能在日常视觉观察与社交互动中积累经验并获取知识。其核心是结构化的长期多模态记忆系统,包含以名称为中心的语义记忆用于组织知识,以及时空联合的情景记忆用于捕捉多模态体验。通过将该记忆系统与基础模型融合,Ella可检索信息支持决策、规划日常活动、建立社交关系,并在与其它智能体共存时自主演化。我们在动态3D开放世界中进行能力评估,让15个智能体持续参与社交活动数日,并通过一系列未见过的受控测试进行评价。实验表明,Ella能有效影响、引领并与其它智能体协作完成任务,展现出通过观察与社交互动高效学习的能力。研究揭示了结构化记忆系统与基础模型结合对推动具身智能发展的变革潜力。更多视频见 https://umass-embodied-agi.github.io/Ella/。

原文摘要 · Abstract (English)

We introduce Ella, an embodied social agent capable of lifelong learning within a community in a 3D open world, where agents accumulate experiences and acquire knowledge through everyday visual observations and social interactions. At the core of Ella's capabilities is a structured, long-term multimodal memory system that stores, updates, and retrieves information effectively. It consists of a name-centric semantic memory for organizing acquired knowledge and a spatiotemporal episodic memory for capturing multimodal experiences. By integrating this lifelong memory system with foundation models, Ella retrieves relevant information for decision-making, plans daily activities, builds social relationships, and evolves autonomously while coexisting with other intelligent beings in the open world. We conduct capability-oriented evaluations in a dynamic 3D open world where 15 agents engage in social activities for days and are assessed with a suite of unseen controlled evaluations. Experimental results show that Ella can influence, lead, and cooperate with other agents well to achieve goals, showcasing its ability to learn effectively through observation and social interaction. Our findings highlight the transformative potential of combining structured memory systems with foundation models for advancing embodied intelligence. More videos can be found at https://umass-embodied-agi.github.io/Ella/.

具身智能社交代理长期记忆3D世界

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。