用开放词汇检测提升零样本视频问答的物体定位与语言对齐能力
ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validation
- 结合思维链与YOLO-World实现动态物体跟踪与语义对齐
- 在NExT-QA等三个基准上达到新最优性能
- 支持跨时段验证，提升多场景视频问答可靠性
近期视频问答（VideoQA）研究引入了基于大模型的智能体、模块化框架与流程化解决方案，取得显著进展。这些系统通过动态智能体和记忆机制分解复杂任务并优化答案。然而，在长时间跨度下追踪物体以实现精准定位，以及基于推理使物体指代与语言模型输出更好对齐方面仍有提升空间，尤其随着新模型在两项能力上的增强。本文提出一个基于大模型的零样本视频问答智能体（ViQAgent），融合思维链框架与基于YOLO-World的开放词汇接地推理，显著提升物体追踪精度与语言对齐能力。该方法在NExT-QA、iVQA和ActivityNet-QA等多个基准上达到新最佳表现。此外，框架支持跨时段接地一致性检查，提升结果准确性与可验证性，适用于多领域视频理解任务。代码已开源：https://github.com/t-montes/viqagent。
原文摘要 · Abstract (English)
Recent advancements in Video Question Answering (VideoQA) have introduced LLM-based agents, modular frameworks, and procedural solutions, yielding promising results. These systems use dynamic agents and memory-based mechanisms to break down complex tasks and refine answers. However, significant improvements remain in tracking objects for grounding over time and decision-making based on reasoning to better align object references with language model outputs, as newer models get better at both tasks. This work presents an LLM-brained agent for zero-shot Video Question Answering (VideoQA) that combines a Chain-of-Thought framework with grounding reasoning alongside YOLO-World to enhance object tracking and alignment. This approach establishes a new state-of-the-art in VideoQA and Video Understanding, showing enhanced performance on NExT-QA, iVQA, and ActivityNet-QA benchmarks. Our framework also enables cross-checking of grounding timeframes, improving accuracy and providing valuable support for verification and increased output reliability across multiple video domains. The code is available at https://github.com/t-montes/viqagent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。