提出流视频问答新任务，精准筛选关键上下文提升推理效率
CogStream: Context-guided Streaming Video Question Answering
- 动态筛选历史上下文，只保留与当前问题相关的信息
- 在真实流场景下实现90%以上准确率，显著优于全量输入方案
- 适合研究多模态推理与视频理解的学者使用
尽管视频大语言模型（Vid-LLMs）在多模态理解方面取得进展，但在流式视频推理中仍面临依赖上下文信息的挑战。现有方法将所有可用的历史上下文输入模型，导致视觉数据处理计算开销大，且无关内容干扰关键信息捕捉。本文提出一项新任务——上下文引导的流式视频推理（CogStream），模拟真实流媒体场景，要求模型从历史信息中识别最相关部分以回答当前问题。为此，我们构建了一个密集标注的数据集，包含大量分层的问答对，通过半自动管道生成。同时提出基线模型 CogReasoner，利用视觉流压缩与历史对话检索有效应对该任务。大量实验证明该方法的有效性。
原文摘要 · Abstract (English)
Despite advancements in Video Large Language Models (Vid-LLMs) improving multimodal understanding, challenges persist in streaming video reasoning due to its reliance on contextual information. Existing paradigms feed all available historical contextual information into Vid-LLMs, resulting in a significant computational burden for visual data processing. Furthermore, the inclusion of irrelevant context distracts models from key details. This paper introduces a challenging task called Context-guided Streaming Video Reasoning (CogStream), which simulates real-world streaming video scenarios, requiring models to identify the most relevant historical contextual information to deduce answers for questions about the current stream. To support CogStream, we present a densely annotated dataset featuring extensive and hierarchical question-answer pairs, generated by a semi-automatic pipeline. Additionally, we present CogReasoner as a baseline model. It effectively tackles this task by leveraging visual stream compression and historical dialogue retrieval. Extensive experiments prove the effectiveness of this method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。