用事件图谱替代原始视频帧,实现长视频问答的高效推理
Semantic Event Graphs for Long-Form Video Question Answering
- 将视频转化为紧凑的语义事件图,以符号化方式保留时间关联
- 仅用3.47千个词元即达65%准确率,比全量输入节省91.4%计算资源
- 适合需要长时序理解的视频问答场景,尤其适用于资源受限部署
长时视频问答对现有视觉语言模型仍是挑战,因其难以在不超预算的情况下处理小时级内容。现有方法通常下采样帧或输入密集视觉嵌入,牺牲时间覆盖换取成本。本文提出语义事件图(SEG),作为视频与语言间的轻量符号接口,用紧凑的时间交互日志替代原始帧。流水线通过YOLOv11检测追踪物体,将空间接近模式转为起止型人-物事件,并组织成时间场景图(TSG)。推理时,查询感知剪枝模块识别锚点实体与语义相关事件,返回小规模子图,经语言化后输入Gemini 2.5 Flash生成答案。在五个YouTube视频(每段含300–500次交互)和120个自动生成的长程问题上,SEG实现65.0%准确率,每查询仅需3.47千词元,接近全日志基线(62.5%准确率,40.39千词元),但词元消耗降低91.4%。仅依赖最后30秒的短上下文基线准确率暴跌至2.5%,凸显显式时间记忆必要性。结果表明,符号化时间图可作为现成视觉语言模型的有效、即插即用记忆层,保持长程推理能力的同时显著降低词元与成本开销。代码、日志及事件提取工具将开源以供复现。
原文摘要 · Abstract (English)
Long-form video question answering remains challenging for modern vision-language models, which struggle to reason over hour-scale footage without exceeding practical token and compute budgets. Existing systems typically downsample frames or feed dense visual embeddings to large-context language models, trading off temporal coverage against cost. We propose Semantic Event Graphs (SEG), a lightweight symbolic interface between video and language that replaces raw frames with compact temporal interaction logs. Our pipeline detects and tracks objects with YOLOv11, converts proximity patterns into START/END human-object events, and organizes them into a Temporal Scene Graph (TSG). At inference time, a query-aware pruning module identifies anchor entities and lexically relevant events, returning only a small subgraph which is verbalized and passed to Gemini 2.5 Flash for answer generation. On five YouTube videos (300-500 interactions each) and 120 automatically generated long-horizon questions, SEG achieves 65.0% accuracy using only 3.47k tokens per query, closely matching a full-log baseline (62.5% at 40.39k tokens) while reducing token usage by 91.4%. A short-context baseline restricted to the last 30 seconds collapses to 2.5% accuracy, underscoring the need for explicit temporal memory. These results show that symbolic temporal graphs can serve as an effective, plug-and-play memory layer for off-the-shelf vision-language models, preserving long-range reasoning ability while making long-form video question answering substantially more token- and cost-efficient. Code, logs, and event-extraction tools will be released for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。