构建复杂场景下音视频说话人追踪数据集，推动真实环境下的多模态感知研究。
AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes
- 设计动态真实场景下的音视频实例分割数据集，支持跨模态追踪
- 在摄像头移动、遮挡等挑战下，现有方法性能显著下降
- 适合研究多模态融合、动态场景理解的学者与开发者
音视频说话人追踪旨在利用听觉和视觉线索定位并跟踪活跃说话人，实现细粒度的人类中心场景理解，对智能视频编辑、监控及人机交互等实际应用至关重要。然而，现有数据集多限于简单或同质化的音视频场景，标注粗糙，导致评估偏向静态的音视频共现关系，难以严格检验复杂动态场景中的时空建模与跨模态推理能力。为此，我们提出AVTrack，一个面向动态真实场景的人类中心音视频实例分割（AVIS）数据集。该数据集涵盖相机运动、视觉遮挡和位置变化等多种复杂条件。在AVTrack上对代表性AVIS方法的评估显示性能显著下降，确立其作为复杂环境中鲁棒人类中心音视频理解的挑战性基准。我们还提供一个简单有效的基线模型以促进后续研究。项目主页：https://FudanCVL.github.io/AVTrack/
原文摘要 · Abstract (English)
Audio-visual speaker tracking aims to localize and track active speakers by leveraging auditory and visual cues, enabling fine-grained, human-centric scene understanding. This capability is essential for real-world applications such as intelligent video editing, surveillance, and human-computer interaction. However, existing datasets are largely limited to simple or homogeneous audio-visual scenes with coarse annotations. Such oversimplified settings bias evaluation toward static audio-visual co-occurrence, rather than rigorously assessing robust spatiotemporal modeling and cross-modal reasoning in complex, dynamic scenes. To address these limitations, we introduce AVTrack, a human-centric audio-visual instance segmentation (AVIS) dataset designed for dynamic real-world scenarios. AVTrack features diverse and challenging conditions, including camera motion, visual occlusions, and position changes. Evaluations of representative AVIS methods on AVTrack reveal substantial performance degradation, establishing AVTrack as a challenging benchmark for robust human-centric audio-visual scene understanding in complex environments. We further provide a simple yet effective baseline to facilitate future research. Project website: https://FudanCVL.github.io/AVTrack/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。