arXiv:2606.17298cs.CV2026-06

通过动作驱动的数字孪生实现手术室视频的推理检索,提升安全事件定位精度。

Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins

论文配图:Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins
图 1 · 摘自论文原文
  • 将手术视频片段转换为动作驱动的数字孪生,按时间分段整合主体-动作-对象三元组。
  • 在276个隐式查询上取得57.6 R@1和77.3 R@5,优于现有方法。
  • 适合需要精准定位复杂手术过程的医疗研究人员与系统开发者。

手术室(OR)中的文本到视频检索是提升手术安全的关键技术,可帮助相关人员检索特定事件的录像。然而,最关键的安全部事件往往不遵循常规结构,因此要发挥其潜力,必须能处理需要推理才能识别正确视频的隐式查询(如“夹闭前的步骤”)。现有方法依赖全局嵌入,无法进行此类推理。本文提出OR3,将视频片段转化为动作驱动的数字孪生(ActDTs),在非重叠时间区间内聚合并发的主体-动作-对象三元组。不同于使用成对编码器进行跨模态匹配,OR3采用基于大模型生成的想象式检索:由大语言模型从查询生成假设的ActDTs,再通过单一编码器进行同模态匹配,并使用专为ActDT设计的难样本训练。最后,通过证据引导的精炼机制,根据与最优候选结果的差异修正生成的ActDTs,以捕捉术式特异性模式。我们在MM-OR数据集上构建了包含276个隐式查询的基准,覆盖四类推理任务,共386个机器人膝关节手术视频。OR3在该任务上达到57.6 R@1和77.3 R@5,显著优于最强基线。结果表明,通过时间动作推理,OR3能够对视觉相似的手术视频片段实现细粒度区分。

原文摘要 · Abstract (English)

Text-to-video retrieval in operating rooms (OR) is an enabling technology for OR safety, as it allows stakeholders to retrieve and inspect recordings of specific events. However, because the most safety-critical events may not follow the common structure, to unlock its full potential text-to-video retrieval must be able to handle implicit queries that require reasoning to identify the right video (e.g., the step right before clipping). However, existing methods rely on global embeddings that cannot reason over such queries. We propose OR3, a text-to-video retrieval method that converts clips into action-driven digital twins (ActDTs), grouping concurrent subject-action-object triplets under non-overlapping temporal intervals. Moreover, rather than cross-modal matching through paired encoders, OR3 performs imagination-based retrieval where an LLM generates hypothetical ActDTs from queries. This enables intra-modal matching via a single encoder trained with ActDT-tailored hard negatives. Finally, evidence-grounded refinement revises imagined ActDTs based on discrepancies with top candidates to capture procedure-specific patterns. We construct a benchmark from MM-OR with 276 implicit queries across four reasoning categories over 386 clips from robotic knee procedures. OR3 achieves 57.6 R@1 and 77.3 R@5, outperforming the strongest baseline. These results demonstrate that OR3 enables fine-grained discrimination between visually similar OR video clips through temporal action reasoning.

视频检索手术室动作推理数字孪生

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。