通过稀疏化视频帧减少视觉令牌，提升自动驾驶多模态大模型推理效率。
Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Driving
- 利用连续视频帧冗余性，自适应筛选关键帧并删除低信息量令牌。
- 在DRAMA和LingoQA上实现33%吞吐量提升、28%内存减少。
- 适合资源受限场景下的自动驾驶多模态模型部署使用。
多模态大语言模型（MLLM）凭借强大的逻辑推理能力，在自动驾驶场景理解中展现出巨大潜力。然而，其庞大的参数量和计算需求常超出车载计算资源限制。主要瓶颈在于为捕捉细粒度与长上下文视觉信息所需大量视觉令牌，导致延迟升高、内存消耗增加。为此，本文提出视频令牌稀疏化（Video Token Sparsification, VTS），利用连续视频帧的内在冗余性，显著减少视觉令牌总数，同时保留关键信息。VTS采用轻量级基于CNN的提案模型，自适应识别关键帧并剔除不相关信息令牌，有效缓解幻觉问题，提升推理吞吐量且不牺牲性能。在DRAMA和LingoQA基准上的全面实验表明，VTS相比基线实现最高33%的推理吞吐量提升和28%的内存占用降低。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have demonstrated remarkable potential for enhancing scene understanding in autonomous driving systems through powerful logical reasoning capabilities. However, the deployment of these models faces significant challenges due to their substantial parameter sizes and computational demands, which often exceed the constraints of onboard computation. One major limitation arises from the large number of visual tokens required to capture fine-grained and long-context visual information, leading to increased latency and memory consumption. To address this issue, we propose Video Token Sparsification (VTS), a novel approach that leverages the inherent redundancy in consecutive video frames to significantly reduce the total number of visual tokens while preserving the most salient information. VTS employs a lightweight CNN-based proposal model to adaptively identify key frames and prune less informative tokens, effectively mitigating hallucinations and increasing inference throughput without compromising performance. We conduct comprehensive experiments on the DRAMA and LingoQA benchmarks, demonstrating the effectiveness of VTS in achieving up to a 33\% improvement in inference throughput and a 28\% reduction in memory usage compared to the baseline without compromising performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。