arXiv:2507.16718cs.CV2025-07被引 3

提出动态时间约束的视频语义分割新任务,让模型根据文本自动判断物体何时该被关注。

Temporally-Constrained Video Reasoning Segmentation and Automated Benchmark Construction

  • 基于文本中隐含的时间逻辑,推断目标物体在视频中的上下文相关时间段。
  • 构建包含52个样本的自动化标注数据集,避免人工标注成本。
  • 适用于手术视频分析等需灵活识别动态对象的场景,适合医疗视觉研究者。

传统视频分割方法受限于预定义类别,无法识别词汇外物体或仅通过复杂文本查询间接提及的物体,限制了其在多变场景中的应用。例如手术室视频分析中,不同医疗系统使用不同流程和器械,难以预先定义所有物体类别。推理分割(RS)虽能通过自然语言查询定位目标,但现有方法假设目标物体在整个视频中持续相关,这在真实场景中不成立——如手术器械仅在特定阶段重要,解剖结构也只在特定时刻关键。本文首次提出时序约束视频推理分割(TC-VRS),要求模型从包含时间推理的文本查询中隐式推断目标物体的上下文相关时段。由于手动标注此类数据成本高昂且难扩展,本文提出一种创新的自动化基准构建方法。最终发布TCVideoRSBenchmark,基于MVOR数据集的52个视频样本,支持时序约束下的视频推理分割研究。

原文摘要 · Abstract (English)

Conventional approaches to video segmentation are confined to predefined object categories and cannot identify out-of-vocabulary objects, let alone objects that are not identified explicitly but only referred to implicitly in complex text queries. This shortcoming limits the utility for video segmentation in complex and variable scenarios, where a closed set of object categories is difficult to define and where users may not know the exact object category that will appear in the video. Such scenarios can arise in operating room video analysis, where different health systems may use different workflows and instrumentation, requiring flexible solutions for video analysis. Reasoning segmentation (RS) now offers promise towards such a solution, enabling natural language text queries as interaction for identifying object to segment. However, existing video RS formulation assume that target objects remain contextually relevant throughout entire video sequences. This assumption is inadequate for real-world scenarios in which objects of interest appear, disappear or change relevance dynamically based on temporal context, such as surgical instruments that become relevant only during specific procedural phases or anatomical structures that gain importance at particular moments during surgery. Our first contribution is the introduction of temporally-constrained video reasoning segmentation, a novel task formulation that requires models to implicitly infer when target objects become contextually relevant based on text queries that incorporate temporal reasoning. Since manual annotation of temporally-constrained video RS datasets would be expensive and limit scalability, our second contribution is an innovative automated benchmark construction method. Finally, we present TCVideoRSBenchmark, a temporally-constrained video RS dataset containing 52 samples using the videos from the MVOR dataset.

视频分割推理分割时序建模医疗影像

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。