用视觉语言模型与几何控制结合，实现零样本室内导航。
OptiSight: Bridging Semantic Reasoning and Geometric Control for Embodied Navigation
- 通过思维链架构融合语义推理与视觉伺服控制
- 在8GB显存下实现跨场景零样本导航，避开障碍物
- 仅关键节点调用大模型，降低计算开销
自主室内导航需兼顾语义理解与精确几何控制。我们提出OptiSight，一种混合框架，将视觉语言模型（VLM）推理与确定性视觉伺服通过有限状态思维链架构结合。Grounded-SAM定位开放词汇目标，相机投影几何将视觉观测转为导航指令，无需稠密地图。VLM仅在关键决策点调用，降低计算开销，几何控制处理连续移动。AI Habitat实验表明，该方法可在多样室内场景中实现可靠零样本导航，包括避障与语义模糊情况，且运行在8~GB显存预算内。源码已公开于https://github.com/avanalperen/OptiSight-Python-Multimodal-CoT-for-Visual-Reasoning。
原文摘要 · Abstract (English)
Autonomous indoor navigation requires both semantic understanding and precise geometric control. We propose OptiSight, a hybrid framework that combines Vision-Language Model reasoning with deterministic visual servoing through a finite-state Chain-of-Thought architecture. Grounded-SAM localizes open-vocabulary targets, while camera projection geometry converts visual observations into navigation commands without requiring dense mapping. The VLM is queried only at key decision points, reducing computational overhead while geometric control handles continuous navigation. Experiments in AI Habitat demonstrate reliable zero-shot navigation across diverse indoor scenarios, including obstacle avoidance and semantic ambiguity, while operating within an 8~GB VRAM budget. The source code is available at https://github.com/avanalperen/OptiSight-Python-Multimodal-CoT-for-Visual-Reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。