用精简运动表示提升长视频理解,效率更高且更准确。
ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
- 用关键帧加运动表示替代全帧输入,降低计算负担。
- 在LongVideoBench等三个基准上显著优于基线模型。
- 适合需要高效处理长视频的多模态应用开发者。
尽管多模态大语言模型(MLLMs)在诸多任务中表现优异,长视频理解仍是重大挑战。传统方法处理完整RGB帧流计算不可行且冗余,因自注意力机制复杂度随序列长度呈二次增长。本文提出ReMoRa,一种直接基于压缩表示处理视频的MLLM。保留稀疏关键帧用于外观建模,时间动态以运动表示编码,无需逐帧解码。该运动表示作为光流紧凑代理,捕捉时序变化。为改善块状运动表示的噪声与低保真问题,引入去噪与细粒度生成模块。模型特征压缩实现线性扩展。在包含LongVideoBench、NExT-QA和MLVU在内的多个长视频理解基准上,ReMoRa显著超越基线方法。
原文摘要 · Abstract (English)
While multimodal large language models (MLLMs) have shown remarkable success across a wide range of tasks, long-form video understanding remains a significant challenge. In this study, we focus on video understanding by MLLMs. This task is challenging because processing a full stream of RGB frames is computationally intractable and highly redundant, as self-attention have quadratic complexity with sequence length. In this paper, we propose ReMoRa, a video MLLM that processes videos by operating directly on their compressed representations. A sparse set of RGB keyframes is retained for appearance, while temporal dynamics are encoded as a motion representation, removing the need for sequential RGB frames. These motion representations act as a compact proxy for optical flow, capturing temporal dynamics without full frame decoding. To refine the noise and low fidelity of block-based motions, we introduce a module to denoise and generate a fine-grained motion representation. Furthermore, our model compresses these features in a way that scales linearly with sequence length. We demonstrate the effectiveness of ReMoRa through extensive experiments across a comprehensive suite of long-video understanding benchmarks. ReMoRa outperformed baseline methods on multiple challenging benchmarks, including LongVideoBench, NExT-QA, and MLVU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。