让大模型和采样器协同进化，高效理解长视频。
MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding
- 通过多视角提问与相似度矩阵，动态筛选关键帧。
- 在多个数据集上提升8.0%准确率，超越最强基线1.1%。
- 适合需要高效处理长视频的视觉问答场景。
高效理解长视频仍是多模态大模型（MLLM）的核心挑战。本文提出MSJoE框架，联合优化MLLM与轻量级关键帧采样器，实现高效长视频理解。该框架基于核心假设：每个问题仅需少量关键帧即可回答。首先生成多个描述不同视觉视角的问题，再与冻结的CLIP模型交互生成查询-帧相似度矩阵；随后由轻量采样器从该矩阵中预测关键帧采样权重，选出紧凑的高信息量帧序列，输入MLLM生成答案。整个过程通过强化学习联合优化，实现查询推理、帧采样与关键帧理解的协同适应。为支持训练，构建了包含2.8K视频与7K问答对的新长视频问答数据集。在VideoMME、LongVideoBench、LVBench和MLVU上的大量实验表明，相较于基础MLLM，MSJoE提升8.0%准确率，较最强基线高出1.1%。
原文摘要 · Abstract (English)
Efficiently understanding long-form videos remains a fundamental challenge for multimodal large language models (MLLMs). In this paper, we present MLLM-Sampler Joint Evolution (MSJoE), a novel framework that jointly evolves the MLLM and a lightweight key-frame sampler for efficient long-form video understanding. MSJoE builds upon a key assumption that only a small subset of key-frames is truly informative for answering each question to a video. Specifically, MSJoE first reasons out several queries, which describe diverse visual perspectives relevant to the question. Then, these queries interact with a frozen CLIP model to produce a query-frame similarity matrix. Finally, a lightweight sampler predicts key-frame sampling weights from this matrix, selecting a compact set of informative frames, which are then fed into the MLLM for answer generation. Both the MLLM and sampler are jointly optimized through reinforcement learning, enabling co-adaptation of query-reasoning, frame-sampling, and key-frame understanding. A new long-video QA dataset containing 2.8K videos with 7K question-answer pairs is collected to support the training process. Extensive experiments on VideoMME, LongVideoBench, LVBench, and MLVU show that MSJoE achieves 8.0\% accuracy gain upon the base MLLM, and 1.1\% higher accuracy than strongest baseline method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。