提出在线多模态视频定位新任务与统一框架,支持文本图像混合查询。
OVG-HQ: Online Video Grounding with Hybrid-modal Queries
- 设计参数化记忆块与跨模态蒸馏,解决在线场景知识遗忘和模态不平衡问题。
- 在自建数据集QVHighlights-Unify上实现领先性能,准确率提升显著。
- 适合需要实时处理图文混合查询的视频理解系统开发者参考。
视频定位(VG)任务旨在根据文本查询定位视频中的特定时刻。然而,传统VG在流式视频或包含视觉线索的查询场景中表现受限。为此,本文提出在线多模态视频定位(OVG-HQ)新任务,支持基于文本、图像、视频片段及其组合的在线片段定位。该任务面临两大挑战:在线设置下上下文有限,训练中主导模态压制弱模态。为此,我们提出OVG-HQ-Unify统一框架,包含参数化记忆块(PMB)以保留先前知识并增强当前决策,以及跨模态蒸馏策略引导非主导模态学习。该设计使单一模型能有效处理混合模态查询。由于缺乏合适数据集,我们构建了QVHighlights-Unify扩展数据集。此外,针对离线指标忽略预测时效性的问题,我们适配引入oR@n、IoU=m和在线平均精度(omAP),同时评估准确率与效率。实验表明,OVG-HQ-Unify优于现有模型,为在线多模态视频定位提供稳健解决方案。代码与数据集见https://github.com/maojiaqi2324/OVG-HQ。
原文摘要 · Abstract (English)
Video grounding (VG) task focuses on locating specific moments in a video based on a query, usually in text form. However, traditional VG struggles with some scenarios like streaming video or queries using visual cues. To fill this gap, we present a new task named Online Video Grounding with Hybrid-modal Queries (OVG-HQ), which enables online segment localization using text, images, video segments, and their combinations. This task poses two new challenges: limited context in online settings and modality imbalance during training, where dominant modalities overshadow weaker ones. To address these, we propose OVG-HQ-Unify, a unified framework featuring a Parametric Memory Block (PMB) that retain previously learned knowledge to enhance current decision and a cross-modal distillation strategy that guides the learning of non-dominant modalities. This design enables a single model to effectively handle hybrid-modal queries. Due to the lack of suitable datasets, we construct QVHighlights-Unify, an expanded dataset with multi-modal queries. Besides, since offline metrics overlook prediction timeliness, we adapt them to the online setting, introducing oR@n, IoU=m, and online mean Average Precision (omAP) to evaluate both accuracy and efficiency. Experiments show that our OVG-HQ-Unify outperforms existing models, offering a robust solution for online, hybrid-modal video grounding. Source code and datasets are available at https://github.com/maojiaqi2324/OVG-HQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。