通过压缩场景区域减少计算量，让自动驾驶VQA更快更高效。
Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression
- 用高阶令牌压缩早期帧，保留最近帧的完整细节
- 在保持性能的同时降低66%计算量（FLOPs）
- 适合对实时性要求高的自动驾驶视觉问答任务
自动驾驶越来越依赖视觉问答（VQA）来理解复杂环境，通过分析视觉输入和文本查询实现。然而，该领域对低延迟和实时处理有严格要求，延迟会直接影响安全。当前主流的视觉语言模型（VLM）虽性能优越，但往往牺牲效率，需处理每帧密集的补丁令牌，导致计算成本过高（FLOPs）和显著推理延迟，尤其在长视频序列中更为严重，限制了其在实时自动驾驶中的部署。为此，我们提出一种面向自动驾驶VQA的高效VLM框架SRC-Pipeline，学习将早期帧的令牌压缩为少量高层令牌，同时保留近期帧的完整补丁令牌。在自动驾驶视频问答任务上的实验表明，该方法可实现66%的FLOPs减少，同时保持相当的性能，使VLM在实时、安全关键的自动驾驶场景中更高效运行。
原文摘要 · Abstract (English)
Autonomous driving increasingly relies on Visual Question Answering (VQA) to enable vehicles to understand complex surroundings by analyzing visual inputs and textual queries. Currently, a paramount concern for VQA in this domain is the stringent requirement for fast latency and real-time processing, as delays directly impact real-world safety in this safety-critical application. However, current state-of-the-art VQA models, particularly large vision-language models (VLMs), often prioritize performance over computational efficiency. These models typically process dense patch tokens for every frame, leading to prohibitive computational costs (FLOPs) and significant inference latency, especially with long video sequences. This focus limits their practical deployment in real-time autonomous driving scenarios. To tackle this issue, we propose an efficient VLM framework for autonomous driving VQA tasks, SRC-Pipeline. It learns to compress early frame tokens into a small number of high-level tokens while retaining full patch tokens for recent frames. Experiments on autonomous driving video question answering tasks show that our approach achieves 66% FLOPs reduction while maintaining comparable performance, enabling VLMs to operate more effectively in real-time, safety-critical autonomous driving settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。