提出视觉感知的专家稀疏路由模型，提升智能体导航定位能力
ViSMoE: Visual-Aware Sparse Mixture-of-Experts for Embodied Referring Expression Grounding
- 用视觉感知路由策略区分视图与物体，针对性处理不同视觉信息
- 在REVERIE和SOON数据集上优于当前最优方法，定位准确率显著提升
- 适合需要精准视觉理解与导航决策的具身智能任务
具身指代表达定位任务要求智能体在真实环境中根据自然语言指令导航并定位目标物体。在此场景中，智能体需在每一步选择一个视角进行导航，并在到达目的地后从候选物体中识别特定目标。然而，以往方法常将视图与物体混用通用视觉编码器处理，导致两者表征模糊。为此，本文提出ViSMoE，通过引入视觉感知的稀疏专家混合路由策略，使智能体能分别处理视图与物体信息，生成更具区分性的视觉表示。在REVERIE和SOON数据集上的实验表明，ViSMoE超越现有最先进方法，验证了该方法的有效性。
原文摘要 · Abstract (English)
Embodied Referring Expression Grounding is the task of enabling an agent to navigate in real environments and to localize a remote object based on natural language instructions. In this scenario, the agent needs to select one view for navigation at each step and identify a specific object among all candidate objects at the destination. However, most of the previous approaches fail to distinguish between views and objects, instead processing them using the vanilla vision encoder, which results in ambiguous representations of both views and objects. To address the above issues, we propose ViSMoE, which equips sparse Mixture-of-Experts with a visual-aware routing policy for the embodied agent. This framework processes different types of visual information specifically, resulting in discriminative visual representations for both views and objects. Experimental results on REVERIE and SOON datasets demonstrate that ViSMoE outperforms the previous state-of-the-art methods, showing the superiority of our proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。