用强化学习压缩视频令牌，95%减量仍保高精度。
MARC: Memory-Augmented RL Token Compression for Efficient Video Understanding
- 先检索关键片段再压缩，用记忆库选重要帧
- 仅用1帧令牌达接近基线性能，减少95%视觉令牌
- 适合实时视频理解，如监控、自动驾驶
大型语言模型的发展为多模态模型奠定了基础。然而，视觉语言模型（VLM）在从图像扩展到视频时仍面临高昂的计算成本，主要源于高帧率和长时长。令牌压缩是一种有前景的解决方案，但现有无训练方法常导致信息丢失和性能下降。为此，我们提出基于记忆增强强化学习的令牌压缩方法（MARC），结合结构化检索与基于强化学习的蒸馏机制。MARC采用“检索-压缩”策略，通过视觉记忆检索器（VMR）选取关键视频片段，并利用压缩组相对策略优化（C-GRPO）框架，将教师模型的推理能力蒸馏至学生模型。在六个视频基准测试上，MARC仅使用1帧的令牌即实现接近基线的准确率，视觉令牌减少95%，GPU内存降低72%，延迟减少23.9%。结果表明，该方法在资源受限场景下具有高效实时视频理解的潜力，适用于视频问答、监控和自动驾驶等应用。
原文摘要 · Abstract (English)
The rapid progress of large language models (LLMs) has laid the foundation for multimodal models. However, visual language models (VLMs) still face heavy computational costs when extended from images to videos due to high frame rates and long durations. Token compression is a promising solution, yet most existing training-free methods cause information loss and performance degradation. To overcome this, we propose \textbf{Memory-Augmented Reinforcement Learning-based Token Compression (MARC)}, which integrates structured retrieval and RL-based distillation. MARC adopts a \textit{retrieve-then-compress} strategy using a \textbf{Visual Memory Retriever (VMR)} to select key clips and a \textbf{Compression Group Relative Policy Optimization (C-GRPO)} framework to distil reasoning ability from a teacher to a student model. Experiments on six video benchmarks show that MARC achieves near-baseline accuracy using only one frame's tokens -- reducing visual tokens by \textbf{95\%}, GPU memory by \textbf{72\%}, and latency by \textbf{23.9\%}. This demonstrates its potential for efficient, real-time video understanding in resource-constrained settings such as video QA, surveillance, and autonomous driving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。