实时视频理解新框架,提速13倍且保持高精度。
RT-NeuS: Towards Real-Time Neuro-Symbolic Video Understanding via Adaptive Temporal Verification
- 自适应采样+批量检测,仅处理关键帧提升效率
- 在长视频上推理速度提升13倍,准确率不降反升
- 适合需要快速响应的复杂视频问答场景
长视频问答(LVQA)需对持续数分钟至数小时的视频回答自然语言问题,要求跨数千帧进行时间推理。标准视觉-语言模型因固定帧预算而强制大幅下采样,导致丢失复杂查询依赖的时间结构。神经符号方法通过将查询分解为原子命题,转换为时序逻辑规范,并使用形式化模型检验来检索满足条件的片段,可在时序复杂基准上提升最高10%的准确率,同时提供可解释性与形式化保证。但构建视频自动机需对每个帧窗口调用视觉-语言模型进行命题锚定,使推理速度比标准提示慢达130倍。本文提出RT-NeuS框架,在保持时序逻辑引导的高准确率和形式化保证的同时,显著缩小延迟差距。该框架引入粗粒度到细粒度的自适应采样机制,识别出少量与查询相关的、视觉差异显著的帧;并通过带键值缓存复用的批量命题检测,在单次前向传播中完成每窗口所有命题评估。我们推导了推理延迟的上界,其取决于视频长度、命题数量和采样密度。在LongVideoBench、Video-MME和MLVU上的实验表明,单张NVIDIA H200 GPU上推理延迟降低高达13倍,同时匹配或超越先前神经符号方法的准确率。
原文摘要 · Abstract (English)
Long-form video question answering (LVQA) requires answering natural-language queries about videos spanning minutes to hours, demanding temporal reasoning across thousands of frames. Standard vision-language models (VLMs) struggle with this task: their fixed frame budgets force aggressive downsampling that misses the temporal structure that complex queries depend on. Neuro-symbolic approaches address this by decomposing queries into atomic propositions, translating them into temporal logic specifications, and applying formal model checking to retrieve segments satisfying the specification. This yields up to 10% higher accuracy on temporally complex benchmarks, with interpretability and formal guarantees. However, constructing the video automaton requires grounding every proposition at every frame window via VLM calls, resulting in up to 130x slower inference than standard VLM prompting. We present RT-NeuS, a framework that preserves the accuracy and formal guarantees of temporal-logic-guided LVQA while closing this latency gap. RT-NeuS introduces coarse-to-fine adaptive sampling to identify the small set of query-relevant, visually distinct frames, and batched proposition detection with KV-cache reuse to evaluate all propositions per window in a single forward pass. We derive latency upper bounds as a function of video length, proposition count, and sampling density. Experiments on LongVideoBench, Video-MME, and MLVU reduce inference latency by up to 13x on a single NVIDIA H200 GPU, while matching or exceeding prior neuro-symbolic accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。