提出自适应跨模态记忆压缩方法，让模型高效理解超长视频并回答复杂问题。
AdaCM$^2$: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction
- 根据视觉与文本关联动态压缩记忆，实现跨模态对齐
- 在LVU数据集上多任务提升4.5%，显存占用降低65%
- 适合处理长视频问答、生成等需要长期记忆的任务
大语言模型（LLMs）的发展推动了视频理解任务的进步，通过将视觉模型与语言模型结合。然而，现有基于LLM的模型（如VideoLLaMA、VideoChat）大多仅能处理短时视频。近期尝试通过提取并压缩视觉特征至固定内存大小来理解长视频，但这些方法仅依赖视觉模态融合视频标记，忽视了视觉与文本查询间的关联，难以有效应对复杂的问答任务。为解决长视频与复杂提示的挑战，我们提出AdaCM²，首次引入自适应跨模态记忆压缩方法，在自回归框架下对视频流进行视频-文本对齐。我们在多个视频理解任务（如视频字幕生成、视频问答、视频分类）上的大量实验表明，AdaCM²在多个数据集上达到当前最优性能，同时显著降低内存消耗。特别地，在LVU数据集上，多任务平均提升4.5%，显存使用减少高达65%。
原文摘要 · Abstract (English)
The advancements in large language models (LLMs) have propelled the improvement of video understanding tasks by incorporating LLMs with visual models. However, most existing LLM-based models (e.g., VideoLLaMA, VideoChat) are constrained to processing short-duration videos. Recent attempts to understand long-term videos by extracting and compressing visual features into a fixed memory size. Nevertheless, those methods leverage only visual modality to merge video tokens and overlook the correlation between visual and textual queries, leading to difficulties in effectively handling complex question-answering tasks. To address the challenges of long videos and complex prompts, we propose AdaCM$^2$, which, for the first time, introduces an adaptive cross-modality memory reduction approach to video-text alignment in an auto-regressive manner on video streams. Our extensive experiments on various video understanding tasks, such as video captioning, video question answering, and video classification, demonstrate that AdaCM$^2$ achieves state-of-the-art performance across multiple datasets while significantly reducing memory usage. Notably, it achieves a 4.5% improvement across multiple tasks in the LVU dataset with a GPU memory consumption reduction of up to 65%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。