让机器人听懂人话并完成室内复杂任务
A Modular Vision-Language-Action Robotics Framework for Indoor Environments
- 模块化设计，分感知与语言双流处理
- 500秒内构建语义体素地图，超时则用部分地图
- 将语言指令与环境上下文结合生成可执行动作
本文提出一个集成系统，用于应对CMU视觉-语言-动作（VLA）挑战，使自主代理能根据自然语言指令执行复杂任务。框架采用模块化架构，协调环境建图、问题处理与导航。系统运行于两条并行流：感知流利用OwlViT嵌入从实时摄像头画面构建语义体素地图；语言流通过视觉-语言模型对用户指令进行分类。建图过程受时间限制，若在500秒探索时限内未完成，则使用部分地图继续推进。分类后的指令在地图的几何与语义上下文中进行语义定位，生成详细提示输入给视觉-语言模型（VLM），从而输出可执行的动作指令，展示了一种有效弥合人类语言与机器人动作之间鸿沟的解决方案。
原文摘要 · Abstract (English)
This paper presents an integrated system for the CMU Vision-Language-Action (VLA) Challenge, designed to enable an autonomous agent to perform complex tasks based on natural language instructions. Our framework employs a modular architecture that orchestrates environment mapping, question processing, and navigation. The system operates in two parallel streams: a perception pipeline that constructs a semantic voxel map from real-time camera feeds using OwlViT embeddings, and a language pipeline that classifies user commands with a Vision-Language Model. The mapping is time-constrained; the system proceeds with a partial map if a 500-second exploration limit is reached. The classified query is then grounded in the geometric and semantic context of the map to generate a detailed prompt for the VLM. This yields an actionable output, demonstrating a capable solution for bridging the gap between human language and robotic action.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。