arXiv:2506.15677cs.AIcs.CL2025-06NeurIPS被引 18

让AI同时懂物理世界和网络信息,实现跨域智能决策

Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence

  • 构建3D环境与网页界面融合的统一仿真平台
  • 提出涵盖烹饪导航等5类任务的跨域评估基准
  • 揭示当前模型在物理数字协同上的显著能力差距

当前AI代理大多孤立:要么处理海量数字信息,要么通过感知与行动与物理世界交互,但极少两者兼具。这种割裂限制了其解决需整合物理与数字智能的任务能力,如根据在线食谱烹饪、利用动态地图导航或用网络知识解读现实地标。本文提出‘具身网络代理’新范式,使代理能流畅衔接身体感知与网络规模推理。为此,我们开发了具身网络代理任务环境,一个将真实3D室内外场景与功能化网页接口紧密结合的统一仿真平台。基于此,构建并发布具身网络代理基准,包含烹饪、导航、购物、旅游和地理定位等多样化任务,均需在物理与数字领域间协调推理,以系统评估跨领域智能水平。实验结果显示,当前顶尖AI系统与人类表现存在显著差距,揭示了具身认知与网络知识获取交叉领域的挑战与机遇。所有数据集、代码及网站已公开于项目主页 https://embodied-web-agent.github.io/。

原文摘要 · Abstract (English)

AI agents today are mostly siloed - they either retrieve and reason over vast amount of digital information and knowledge obtained online; or interact with the physical world through embodied perception, planning and action - but rarely both. This separation limits their ability to solve tasks that require integrated physical and digital intelligence, such as cooking from online recipes, navigating with dynamic map data, or interpreting real-world landmarks using web knowledge. We introduce Embodied Web Agents, a novel paradigm for AI agents that fluidly bridge embodiment and web-scale reasoning. To operationalize this concept, we first develop the Embodied Web Agents task environments, a unified simulation platform that tightly integrates realistic 3D indoor and outdoor environments with functional web interfaces. Building upon this platform, we construct and release the Embodied Web Agents Benchmark, which encompasses a diverse suite of tasks including cooking, navigation, shopping, tourism, and geolocation - all requiring coordinated reasoning across physical and digital realms for systematic assessment of cross-domain intelligence. Experimental results reveal significant performance gaps between state-of-the-art AI systems and human capabilities, establishing both challenges and opportunities at the intersection of embodied cognition and web-scale knowledge access. All datasets, codes and websites are publicly available at our project page https://embodied-web-agent.github.io/.

具身智能多模态推理人机交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。