用逻辑推理精确定位长视频关键片段,提升复杂问题回答准确率
NeuS-QA: Grounding Long-Form Video Understanding in Temporal Logic and Neuro-Symbolic Reasoning
- 将问题转为时间逻辑,通过模型检验筛选符合要求的视频片段
- 在LongVideoBench和CinePile上性能提升超10%,尤其擅长因果与多步推理
- 无需训练或微调,可直接接入现有视觉语言模型,适合需要可解释性的研究
尽管视觉语言模型(VLMs)在单图或短视频任务中表现优异,但在长视频问答(LVQA)上仍面临多步时间推理的挑战。传统方法对长视频均匀采样后输入VLM,导致令牌开销大,需大幅降采样,丢失细粒度视觉结构与关键时间线索。近期方法虽尝试改进,但缺乏显式的时间关系建模机制,也无法保证采样上下文满足问题所需的组合或因果逻辑。为此,我们提出NeuS-QA,一种无需训练、即插即用的神经符号推理框架。该方法将自然语言问题转化为时间逻辑规范,构建视频自动机描述帧级事件演化,并通过模型检验识别满足逻辑条件的视频片段。仅将这些逻辑验证后的片段输入VLM,从而提升可解释性、减少幻觉,并实现组合推理,无需修改或微调模型。在LongVideoBench和CinePile基准上的实验表明,该方法性能提升超过10%,尤其在事件排序、因果关系和多步推理类问题上表现显著。
原文摘要 · Abstract (English)
While vision-language models (VLMs) excel at tasks involving single images or short videos, they still struggle with Long Video Question Answering (LVQA) due to its demand for complex multi-step temporal reasoning. Vanilla approaches, which simply sample frames uniformly and feed them to a VLM along with the question, incur significant token overhead. This forces aggressive downsampling of long videos, causing models to miss fine-grained visual structure, subtle event transitions, and key temporal cues. Recent works attempt to overcome these limitations through heuristic approaches; however, they lack explicit mechanisms for encoding temporal relationships and fail to provide any formal guarantees that the sampled context actually encodes the compositional or causal logic required by the question. To address these foundational gaps, we introduce NeuS-QA, a training-free, plug-and-play neuro-symbolic pipeline for LVQA. NeuS-QA first translates a natural language question into a logic specification that models the temporal relationship between frame-level events. Next, we construct a video automaton to model the video's frame-by-frame event progression, and finally employ model checking to compare the automaton against the specification to identify all video segments that satisfy the question's logical requirements. Only these logic-verified segments are submitted to the VLM, thus improving interpretability, reducing hallucinations, and enabling compositional reasoning without modifying or fine-tuning the model. Experiments on the LongVideoBench and CinePile LVQA benchmarks show that NeuS-QA significantly improves performance by over 10%, particularly on questions involving event ordering, causality, and multi-step reasoning. We open-source our code at https://utaustin-swarmlab.github.io/NeuS-QA/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。