arXiv:2504.15918cs.CVcs.AI2025-04被引 2

让AI通过提问逐步定位视频答案,提升指令理解准确性。

Ask2Loc: Learning to Locate Instructional Visual Answers by Asking Questions

论文配图:Ask2Loc: Learning to Locate Instructional Visual Answers by Asking Questions
图 1 · 摘自论文原文
  • 通过交互式提问澄清用户意图,解决语义模糊问题。
  • 在三个数据集上实现最高14.91的mIoU提升,显著优于传统方法。
  • 适合需要精准定位视频内容的研究者与开发者使用。

在教学视频中定位特定片段是获取指导信息的有效方式。通常,根据语言解释和视觉演示获取视频片段的任务称为视觉答案定位(VAL)。然而,用户在使用系统时往往需多次交互才能获得符合预期的答案。在此过程中,人类通过自我提问加深对视频内容的理解,从而准确定位。为此,我们提出新任务In-VAL,模拟人与视频间多轮交互以获取视觉答案的过程。In-VAL任务需解决多个语义鸿沟问题:1)输入问题中用户意图模糊;2)视频字幕语言不完整;3)视频片段内容碎片化。为此,我们提出Ask2Loc框架,通过提问解决In-VAL问题。该框架包含三个关键模块:1)聊天模块用于细化初始问题,揭示明确意图;2)重写模块生成流畅语言并构建完整描述;3)搜索模块扩展局部上下文,提供整合内容。我们在三个重构的In-VAL数据集上进行了大量实验。相比传统端到端及两阶段方法,所提Ask2Loc在In-VAL任务上性能提升高达14.91(mIoU)。代码与数据集可访问:https://github.com/changzong/Ask2Loc。

原文摘要 · Abstract (English)

Locating specific segments within an instructional video is an efficient way to acquire guiding knowledge. Generally, the task of obtaining video segments for both verbal explanations and visual demonstrations is known as visual answer localization (VAL). However, users often need multiple interactions to obtain answers that align with their expectations when using the system. During these interactions, humans deepen their understanding of the video content by asking themselves questions, thereby accurately identifying the location. Therefore, we propose a new task, named In-VAL, to simulate the multiple interactions between humans and videos in the procedure of obtaining visual answers. The In-VAL task requires interactively addressing several semantic gap issues, including 1) the ambiguity of user intent in the input questions, 2) the incompleteness of language in video subtitles, and 3) the fragmentation of content in video segments. To address these issues, we propose Ask2Loc, a framework for resolving In-VAL by asking questions. It includes three key modules: 1) a chatting module to refine initial questions and uncover clear intentions, 2) a rewriting module to generate fluent language and create complete descriptions, and 3) a searching module to broaden local context and provide integrated content. We conduct extensive experiments on three reconstructed In-VAL datasets. Compared to traditional end-to-end and two-stage methods, our proposed Ask2Loc can improve performance by up to 14.91 (mIoU) on the In-VAL task. Our code and datasets can be accessed at https://github.com/changzong/Ask2Loc.

视频定位交互式问答视觉理解多轮对话

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。