将大模型空间推理能力压缩到40亿参数小模型,实现机器人本地低延迟导航。
LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation

- 用500条推理轨迹微调40亿参数小模型,迁移大模型的语义理解能力
- 在HM3D数据集上达成34.5%成功率,接近云端大模型表现
- 通过序列压缩与量化,本地部署延迟降低82.8%,适合移动机器人使用
视觉语言模型(VLMs)在机器人领域展现出强大环境感知能力,可支持如目标导航(ObjectNav)等开放词汇任务。然而其计算开销通常需依赖云端执行,难以在资源受限的机器人上实现低延迟推理。为此,本文提出一种知识蒸馏策略,将前沿大模型的空间-语义推理能力迁移到仅40亿参数的轻量级本地VLM中,实现在嵌入式GPU设备(如Jetson Orin)上的边缘部署。首先,基于Claude Sonnet 4.6构建基于场景图(SG)的先进方法,在HM3D OVON基准上取得39.7%的成功率(SR)。随后,仅用500条前沿模型推理轨迹微调Qwen3.5-4B,即实现34.5%的SR,显著缩小与大模型性能差距。最后,引入带有令牌生成(TG)正则化的E-RLVR框架,压缩输出序列长度以适应物理部署,使TG开销降低72.1%,延迟减少71.8%。结合量化技术,整体推理延迟降低82.8%,在不明显牺牲性能的前提下,为移动机器人提供了可行的本地化、低延迟VLM执行范式。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context, serving as a catalyst for open-vocabulary tasks like ObjectNav. Yet, their computational footprint typically confines them to cloud execution, hindering low-latency inference with local deployment on resource-constrained robots. To address this challenge, we present a distillation strategy that transfers complex spatial-semantic reasoning from large frontier models into a lightweight, 4B-parameter local VLM for edge execution on embedded GPU devices (e.g., Jetson Orin). We first establish a State of the Art (SotA), Scene Graph (SG)-based pipeline using Claude Sonnet 4.6, achieving a 39.7% Success Rate (SR) on the HM3D OVON benchmark. We then demonstrate that fine-tuning Qwen3.5-4B on just 500 frontier reasoning traces effectively enables navigation capabilities, yielding a SR of 34.5%, narrowing the gap to the performance of large cloud models. Finally, we introduce E-RLVR with Token Generation (TG) regularization to compress output sequence lengths for physical deployment while grounding the agent in its task. This downstream optimization reduces TG overhead by 72.1% and latency by 71.8%. Combined with quantization, this joint strategy yields a cumulative 82.8% reduction in overall inference latency without significantly sacrificing performance, presenting a viable paradigm for local, low-latency VLM execution on mobile robots.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。