通过智能筛选减少视觉语言导航的计算开销，不损失效果还省超一半算力。
Walk and Read Less: Improving the Efficiency of Vision-and-Language Navigation via Tuning-Free Multimodal Token Pruning
- 根据导航能力预先区分图像前景与背景，只对背景令牌剪枝。
- 在标准数据集上剪枝后仍保持高成功率，算力消耗降低50%以上。
- 适合资源受限场景下的实时导航系统部署，无需重新训练模型。
大模型在视觉语言导航（VLN）任务中表现优异，但在资源受限环境下运行成本高。令牌剪枝可通过减小输入规模提升效率，但现有方法忽略VLN特有的挑战：剪枝导致的信息丢失可能因路径变长而增加实际计算开销。为此，我们提出导航感知剪枝（NAP），利用导航特性简化剪枝过程——先将图像视图按可否导航方向预分类为前景与背景；再用大语言模型提取导航相关指令。随后仅对背景令牌进行剪枝，并通过移除低重要性导航节点来避免回溯，从而控制路径长度。在标准VLN基准测试中，NAP显著优于已有方法，在保留更高成功率的同时，节省超过50%的浮点运算量（FLOPS）。
原文摘要 · Abstract (English)
Large models achieve strong performance on Vision-and-Language Navigation (VLN) tasks, but are costly to run in resource-limited environments. Token pruning offers appealing tradeoffs for efficiency with minimal performance loss by reducing model input size, but prior work overlooks VLN-specific challenges. For example, information loss from pruning can effectively increase computational cost due to longer walks. Thus, the inability to identify uninformative tokens undermines the supposed efficiency gains from pruning. To address this, we propose Navigation-Aware Pruning (NAP), which uses navigation-specific traits to simplify the pruning process by pre-filtering tokens into foreground and background. For example, image views are filtered based on whether the agent can navigate in that direction. We also extract navigation-relevant instructions using a Large Language Model. After filtering, we focus pruning on background tokens, minimizing information loss. To further help avoid increases in navigation length, we discourage backtracking by removing low-importance navigation nodes. Experiments on standard VLN benchmarks show NAP significantly outperforms prior work, preserving higher success rates while saving more than 50% FLOPS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。