提出新采样方法，让视频大模型更准定位关键片段。
GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
- 根据查询动态筛选关键视觉令牌，避免均匀采样漏掉重要帧。
- 在三个基准上提升7.7点mIoU和12.0点mAP，定位更精准。
- 适合需要高精度视频时序定位的研究与应用，如智能检索。
视频时序定位（VTG）是视频理解的关键任务，也是扩展视频大语言模型（Vid-LLMs）应用范围的核心能力。然而，现有Vid-LLMs依赖均匀帧采样提取视频信息，导致关键帧分布稀疏，丢失重要时序线索。为此，我们提出基于语义引导的视觉令牌采样方法（GroundVTS），聚焦最具信息量的时序段落。GroundVTS采用细粒度、查询驱动机制，在输入大语言模型前过滤视觉令牌，有效保留时空信息并维持时序连贯性。此外，引入渐进式优化策略，使大语言模型能适应视觉特征的非均匀分布，增强对时序依赖建模能力，实现精确视频定位。我们在三个标准VTG基准上全面评估，结果表明，该方法在时刻检索任务中实现7.7点的mIoU提升，在亮点检测任务中实现12.0点的mAP提升。代码已开源：https://github.com/Florence365/GroundVTS。
原文摘要 · Abstract (English)
Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at https://github.com/Florence365/GroundVTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。