用链式采样优化长视频理解，自动选关键画面。
CoS: Chain-of-Shot Prompting for Long Video Understanding
- 将画面选择转化为任务相关的视觉提示优化。
- 在五个数据集上提升长视频理解准确率，效果稳定。
- 适合需要精准理解长视频的场景，如智能监控、教育分析。
多模态大模型（MLLM）在处理长视频时受限于视觉标记过多，超出模型上下文长度，导致大量无关画面填充。如何选择关键画面仍是未解难题：稀疏采样易遗漏细节，全量采样则使模型被冗余内容淹没，造成理解错误。为此，我们提出链式采样提示（CoS）。核心思想是将画面选择视为测试时的视觉提示优化，通过优化画面与任务语义的对齐关系，自适应地选择关键画面。CoS包含两部分：(1) 二值视频摘要机制，实现伪时间定位，生成二进制编码识别任务相关画面；(2) 视频协同推理模块，利用该编码将相关正样本与无关负样本配对学习，强化对齐。最终将优化后的画面嵌入原视频，引导模型聚焦关键信息。在三个基线和五个数据集上的实验验证了CoS的有效性与适应性。代码已公开：https://lwpyh.github.io/CoS。
原文摘要 · Abstract (English)
Multi-modal Large Language Models (MLLMs) struggle with long videos due to the need for excessive visual tokens. These tokens exceed massively the context length of MLLMs, resulting in filled by redundant task-irrelevant shots. How to select shots is an unsolved critical problem: sparse sampling risks missing key details, while exhaustive sampling overwhelms the model with irrelevant content, leading to video misunderstanding. To solve this problem, we propose Chain-of-Shot prompting (CoS). The key idea is to frame shot selection as test-time visual prompt optimisation, choosing shots adaptive to video understanding semantic task by optimising shots-task alignment. CoS has two key parts: (1) a binary video summary mechanism that performs pseudo temporal grounding, discovering a binary coding to identify task-relevant shots, and (2) a video co-reasoning module that deploys the binary coding to pair (learning to align) task-relevant positive shots with irrelevant negative shots. It embeds the optimised shot selections into the original video, facilitating a focus on relevant context to optimize long video understanding. Experiments across three baselines and five datasets demonstrate the effectiveness and adaptability of CoS. Code given in https://lwpyh.github.io/CoS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。