arXiv:2409.01071cs.CVcs.CL2024-09ICCV被引 18

用记忆桥技术实现超长视频高效理解,精度与扩展性兼备。

VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridges

论文配图:VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridges
图 1 · 摘自论文原文
  • 引入循环记忆桥和时间记忆标记,连续编码长视频
  • 在4个VideoQA任务上超越现有模型4.2分,极端长度下仍稳定
  • 单卡处理320帧,适合学术界低成本研究长视频

近期大规模视频-语言模型在实时规划和细节交互方面展现出巨大潜力,但其高计算成本和标注数据稀缺限制了实际应用。本文提出VideoLLaMB,一种新颖高效的长视频理解框架,利用循环记忆桥和时间记忆标记,实现全视频序列的无缝编码并保持语义连贯性。核心是SceneTiling算法,将视频分割为连贯语义单元,支持跨任务鲁棒理解且无需额外训练。VideoLLaMB在四个VideoQA基准上领先现有模型4.2分,在第一人称规划任务上提升2.06分。在极端视频长度扩展(达8倍)下表现优异,并在新提出的针堆中找针(NIAVH)细粒度帧检索任务中表现突出。具备线性GPU内存增长特性,仅用单块Nvidia A100 GPU即可处理最多320帧,尽管训练仅基于16帧,实现了准确率、可扩展性与成本效益的空前平衡,极大提升了学术界的可及性与实用性。

原文摘要 · Abstract (English)

Recent advancements in large-scale video-language models have shown significant potential for real-time planning and detailed interactions. However, their high computational demands and the scarcity of annotated datasets limit their practicality for academic researchers. In this work, we introduce VideoLLaMB, a novel and efficient framework for long video understanding that leverages recurrent memory bridges and temporal memory tokens to enable seamless encoding of entire video sequences with preserved semantic continuity. Central to our approach is a SceneTiling algorithm that segments videos into coherent semantic units, facilitating robust understanding across tasks without requiring additional training. VideoLLaMB achieves state-of-the-art performance, surpassing existing models by 4.2 points on four VideoQA benchmarks and by 2.06 points on egocentric planning tasks. Notably, it maintains strong performance under extreme video length scaling (up to 8 times) and excels at fine-grained frame retrieval on our proposed Needle in a Video Haystack (NIAVH) benchmark. With linear GPU memory scaling, VideoLLaMB processes up to 320 frames using a single Nvidia A100 GPU, despite being trained on only 16 frames-offering an unprecedented balance of accuracy, scalability, and cost-effectiveness. This makes it highly accessible and practical for the academic community.

视频理解长视频记忆机制高效模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。