通过动态剪枝视觉标记,让视觉语言动作模型更高效地处理自动驾驶中的时序信息。
ETA-VLA: Efficient Token Adaptation via Temporal Fusion and Intra-LLM Sparsification for Vision-Language-Action Models
- 用文本引导的评分与多样性保留策略,动态筛选关键视觉标记。
- 在NAVSIM v2上剪掉85%视觉标记,推理耗能降低61%,精度仍保持94%。
- 适合追求高效自动驾驶模型部署的研究者和工程师。
将视觉-语言-动作(VLA)模型融入自动驾驶系统,可统一解析复杂场景并执行控制指令。但为实现准确的时序推理,需处理历史多视角图像帧,带来严重计算负担,主要源于大语言模型(LLM)自注意力机制的二次方复杂度。为此,我们提出ETA-VLA:一种面向VLA模型的高效标记适应框架。该框架处理过去n帧多视角图像,引入新型内部LLM稀疏聚合器(ILSA)。受人类驾驶员注意力分配启发,ILSA基于文本查询与时间一致性,动态识别并剔除冗余视觉标记。具体采用文本引导评分机制结合保多样性的稀疏化策略,选取关键标记子集,确保对驾驶场景的全面感知。在NAVSIM v2上的大量实验表明,ETA-VLA达到与先进基线相当的驾驶性能,同时计算量减少约32%。值得注意的是,该方法剪除85%视觉标记,推理FLOPs降低61%,仍保持原模型94%的准确率。
原文摘要 · Abstract (English)
The integration of Vision-Language-Action (VLA) models into autonomous driving systems offers a unified framework for interpreting complex scenes and executing control commands. However, the necessity to incorporate historical multi-view frames for accurate temporal reasoning imposes a severe computational burden, primarily driven by the quadratic complexity of self-attention mechanisms in Large Language Models (LLMs). To alleviate this bottleneck, we propose ETA-VLA, an Efficient Token Adaptation framework for VLA models. ETA-VLA processes the past $n$ frames of multi-view images and introduces a novel Intra-LLM Sparse Aggregator (ILSA). Drawing inspiration from human driver attention allocation, ILSA dynamically identifies and prunes redundant visual tokens guided by textual queries and temporal consistency. Specifically, we utilize a text-guided scoring mechanism alongside a diversity-preserving sparsification strategy to select a sparse subset of critical tokens, ensuring comprehensive awareness of the driving scene. Extensive experiments on the NAVSIM v2 demonstrate that ETA-VLA achieves driving performance comparable to state-of-the-art baselines while reducing computational FLOPs by approximately 32\%. Notably, our method prunes 85% of visual tokens and reduces inference FLOPs by 61\%, but still retaining 94% of the original accuracy on the NAVSIM v2 benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。