用自回归选图法减少视频冗余，让大模型更快更准看长高清视频。
Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing
- 先用自回归机制选出关键图像块，再交由视觉模型处理
- 压缩90%以上视觉标记，提速达19倍，支持1000帧4K视频
- 适合需要高效处理长视频的多模态大模型研究者
多模态大语言模型虽已推动通用视频理解，但面对长时、高分辨率视频时仍受限于对所有像素等量处理，存在显著时空冗余。我们提出AutoGaze——一个轻量级模块，在视觉变压器（ViT）或大模型处理前剔除冗余图像块。该模块通过下一个词预测与强化学习训练，自回归地选择最小规模的多尺度图像块集合，可在用户指定误差阈值内重建视频，消除冗余同时保留信息。实验证明，AutoGaze可将视觉标记数减少4倍至100倍，使ViT和多模态大模型加速达19倍，首次实现对1000帧4K分辨率视频的建模，并在视频基准测试中取得优异表现（如VideoMME达67.0%）。此外，我们构建了首个高分辨率长视频问答基准HLVid，包含5分钟4K视频；经AutoGaze扩展的模型相比基线提升10.1%，优于此前最佳多模态模型4.5%。
原文摘要 · Abstract (English)
Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。