用多视频协作推理提升视频大模型准确性，避免幻觉。
Enhancing Video Large Language Models with Structured Multi-Video Collaborative Reasoning
- 将视频转为时空图结构，高效表达知识。
- 融合多视频信息，生成增强的图节点令牌。
- 适合需要高精度视频理解的场景，如智能客服、内容审核。
尽管视频语言模型发展迅速，但单一视频固有的时空不完整性导致推理时出现幻觉和错误。通过引入相关多视频可提升推理性能，但视频标记数量庞大且冗余，直接输入大模型可能适得其反。为此，我们提出一种多视频协作框架：首先构建视频结构模块，将视频知识表示为时空图；再设计图融合模块，将相关视频中的有价值信息融入增强的图节点标记；最后构建精细的多视频结构化提示，整合图、视觉与文本标记作为大模型输入。大量实验证明该框架有效，展现出推动视频语言模型发展的潜力。代码将在 https://github.com/ziHoHe/SMV-CR 开源。
原文摘要 · Abstract (English)
Despite the prosperity of the video language model, the current pursuit of comprehensive video reasoning is thwarted by the inherent spatio-temporal incompleteness within individual videos, resulting in hallucinations and inaccuracies. A promising solution is to augment the reasoning performance with multiple related videos. However, video tokens are numerous and contain redundant information, so directly feeding the relevant video data into a large language model to enhance responses could be counterproductive. To address this challenge, we propose a multi-video collaborative framework for video language models. For efficient and flexible video representation, we establish a Video Structuring Module to represent the video's knowledge as a spatio-temporal graph. Based on the structured video representation, we design the Graph Fusion Module to fuse the structured knowledge and valuable information from related videos into the augmented graph node tokens. Finally, we construct an elaborate multi-video structured prompt to integrate the graph, visual, and textual tokens as the input to the large language model. Extensive experiments substantiate the effectiveness of our framework, showcasing its potential as a promising avenue for advancing video language models. Code will be open-sourced at https://github.com/ziHoHe/SMV-CR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。