基于通义千问多模态大模型的视觉语言导航系统，实现高精度动作预测。
PGN: Design and Implementation of a Vision-Language Navigation System Based on Pangu Multimodal Foundation Model
- 采用双阶段训练：先对齐视觉与语言编码器，再适配专家导航轨迹。
- 在500条保留轨迹上达62.29%的归一化动作匹配率，非空率100%。
- 专为昇腾910B硬件优化，支持混合精度与分布式训练，适合工业级部署。
视觉语言导航（VLN）要求一个具身智能体根据自然语言指令，从时序视觉观测中预测动作。将多模态大语言模型应用于VLN需解决视觉-语言对齐、紧凑时序输入、动作空间锚定及目标硬件上的稳定训练问题。本技术报告介绍基于OpenPangu-7B构建的离线动作预测系统PGN（Pangu Navigator）。训练分为两阶段：第一阶段，通过训练Q-Former和两层MLP投影器，对齐冻结的EVA-ViT-G/14视觉编码器与冻结的语言主干；第二阶段，利用五观察窗口、随轮次变化的时序采样策略以及推理-动作输出格式，适配对齐模型至专家导航轨迹，此阶段冻结视觉路径，仅更新三个结构化标记嵌入与LoRA适配器。实现结合混合精度计算、选择性FP32计算与DeepSpeed ZeRO-2，在八块昇腾910B NPU上完成。在500条保留专家轨迹的教师强制、开环评估下，V9版本报告62.29%的归一化动作匹配率（NAM）与100.00%的非空率（NER）。这些指标衡量离线专家动作对齐性能，而非闭环导航成功；误差累积、路径效率与目标达成仍为后续工作。
原文摘要 · Abstract (English)
Vision-Language Navigation (VLN) requires an embodied agent to interpret a natural-language instruction and predict actions from temporally ordered visual observations. Adapting a multimodal large language model to VLN requires visual-language alignment, compact temporal inputs, action-space grounding, and stable training on the target hardware. This technical report presents PGN (Pangu Navigator), an offline VLN action-prediction system built on OpenPangu-7B. Training proceeds in two stages. First, PGMM aligns a frozen EVA-ViT-G/14 vision encoder with the frozen language backbone by training a Q-Former and a two-layer MLP projector. Second, PGN adapts the aligned model to expert navigation trajectories using five-observation windows, epoch-dependent temporal sampling, and a reasoning-then-action output format; this stage freezes the aligned visual pathway and updates three structural-token embeddings and LoRA adapters. The implementation combines mixed-precision computation, selective FP32 computation, and DeepSpeed ZeRO-2 on eight Ascend 910B NPUs. Under teacher-forced, open-loop evaluation on 500 held-out expert trajectories, V9 reports a 62.29% Normalized Action Match (NAM) and a 100.00% Non-empty Rate (NER). These metrics quantify offline expert-action alignment rather than closed-loop navigation success; evaluating error accumulation, path efficiency, and goal completion remains future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。