arXiv:2605.28604cs.CVcs.AI2026-05

通过时空线索识别视频中关键人物,解决重要性随时间变化的问题。

Mining Multi-Modality Spatio-Temporal Cues for Video Important Person Identification

论文配图:Mining Multi-Modality Spatio-Temporal Cues for Video Important Person Identification
图 1 · 摘自论文原文
  • 融合多模态时空信息,构建分层推理框架
  • 在9249段视频上实现67.3%识别准确率,优于现有模型
  • 提供可解释文本理由,适合智能剪辑与安防场景

视频中关键人物的识别对自动化视频编辑和智能监控至关重要。现有方法多依赖静态图像和即时视觉线索,忽视了视频中丰富的时空信息,导致时间重要性偏移(TIS)现象:早期帧中重要的个体在考虑完整时序后可能被重新评估。为此,我们提出视频关键人物(VIP)识别任务,旨在自动识别视频中最具影响力的人物,并生成文本解释。我们构建了Temporal-VIP数据集,包含11个类别、9,249个视频片段,配有对齐的重要性解释。为缓解TIS,提出VIP-Net框架,包含社会线索编码器(SCE)提取多模态时空特征,时序重要性校正器(TIR)进行层次化特征融合与跨模态对齐,以及基于排序的VIP推理模块。实验表明,VIP-Net达到67.3%的准确率,显著优于现有模型(37.5%-53.9%),并通过特征引导的大语言模型优化,实现0.63的平均解释相似度。代码与数据已公开于https://huggingface.co/datasets/yml2002/Temporal-VIP。

原文摘要 · Abstract (English)

Identifying key individuals in video scenes is essential for applications such as automated video editing and intelligent surveillance. Current methods primarily focus on static images and immediate visual cues, overlooking the rich spatio-temporal information in videos. This leads to the phenomenon of Temporal Importance Shift (TIS), wherein individuals deemed significant in early frames may be demoted as the entire temporal context is considered. To address this, we introduce the Video Important Person (VIP) identification task, aimed at automatically identifying the most influential individuals in videos while providing textual rationales. We present Temporal-VIP, a large-scale rationale-annotated dataset consisting of 9,249 video segments across 11 categories with aligned importance rationales. To mitigate TIS, we develop the VIP-Net framework, which includes a Social Cue Encoder (SCE) for extracting multi-modal spatio-temporal cues, a Temporal Importance Rectifier (TIR) for hierarchical cue fusion and cross-modal alignment, and VIP Inference for ranking individuals. Experimental results show that VIP-Net achieves 67.3% accuracy, significantly outperforming state-of-the-art models (37.5%-53.9%) and yielding a mean rationale similarity of 0.63 to ground truth through feature-guided LLM refinement. The dataset and code are available at https://huggingface.co/datasets/yml2002/Temporal-VIP.

视频理解关键人物识别多模态可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。