用两阶段知识蒸馏，让小模型达到大模型导航水平
MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillation
- 分阶段蒸馏：先学通用视觉语言知识，再专攻导航任务
- 参数量仅12%却在R2R/REVERIE上媲美大模型
- 适合资源受限的机器人导航部署场景
近年来，具身人工智能（Embodied AI）发展迅速，但模型规模增大与具身平台计算能力有限之间存在矛盾。本文聚焦具身AI核心任务——视觉语言导航（VLN），提出一种两阶段知识蒸馏框架，训练出轻量级学生模型MiniVLN，展现了蒸馏技术在构建高效模型中的巨大潜力。该方法在预训练阶段捕捉细粒度知识，在微调阶段学习导航特定知识。实验表明，相较于单阶段蒸馏，两阶段方法能更有效缩小教师模型与学生模型间的性能差距。在公开的R2R和REVERIE基准测试中，MiniVLN仅需教师模型约12%的参数量，即可实现相当的导航性能。
原文摘要 · Abstract (English)
In recent years, Embodied Artificial Intelligence (Embodied AI) has advanced rapidly, yet the increasing size of models conflicts with the limited computational capabilities of Embodied AI platforms. To address this challenge, we aim to achieve both high model performance and practical deployability. Specifically, we focus on Vision-and-Language Navigation (VLN), a core task in Embodied AI. This paper introduces a two-stage knowledge distillation framework, producing a student model, MiniVLN, and showcasing the significant potential of distillation techniques in developing lightweight models. The proposed method aims to capture fine-grained knowledge during the pretraining phase and navigation-specific knowledge during the fine-tuning phase. Our findings indicate that the two-stage distillation approach is more effective in narrowing the performance gap between the teacher model and the student model compared to single-stage distillation. On the public R2R and REVERIE benchmarks, MiniVLN achieves performance on par with the teacher model while having only about 12% of the teacher model's parameter count.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。