实时检测用户查询事件的起始点，提升视觉语言模型响应速度。
Streaming Detection of Queried Event Start
- 采用适配器结构实现图像到视频的高效迁移学习。
- 在Ego4D数据集上实现低延迟高精度的事件起始检测。
- 适合需要实时交互的机器人与增强现实应用。
机器人、自动驾驶、增强现实等具身视觉应用需快速响应用户定义的实时事件。本文提出多模态视频理解新任务——流式查询事件起始检测（SDQES），目标是在自然语言查询描述下，以高准确率和低延迟识别复杂事件的开始时刻。基于Ego4D数据集构建新基准，并设计任务专用评估指标，研究第一人称视频中多样化事件的流式多模态检测。受NLP与视频任务中参数高效微调方法启发，提出基于适配器的基线模型，支持图像到视频的迁移学习，实现高效的在线视频建模。在短片段与无修剪视频设置下，评估三种视觉-语言主干网络与三种适配器架构的性能。
原文摘要 · Abstract (English)
Robotics, autonomous driving, augmented reality, and many embodied computer vision applications must quickly react to user-defined events unfolding in real time. We address this setting by proposing a novel task for multimodal video understanding-Streaming Detection of Queried Event Start (SDQES). The goal of SDQES is to identify the beginning of a complex event as described by a natural language query, with high accuracy and low latency. We introduce a new benchmark based on the Ego4D dataset, as well as new task-specific metrics to study streaming multimodal detection of diverse events in an egocentric video setting. Inspired by parameter-efficient fine-tuning methods in NLP and for video tasks, we propose adapter-based baselines that enable image-to-video transfer learning, allowing for efficient online video modeling. We evaluate three vision-language backbones and three adapter architectures on both short-clip and untrimmed video settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。