首个在线音视频实例分割框架,让系统实时追踪发声目标。
SeaVIS: Sound-Enhanced Association for Online Audio-Visual Instance Segmentation
- 用因果交叉注意力融合视觉与音频历史,支持流式处理
- 在AVISeg数据集上准确率超越现有方法,实现实时推理
- 结合声音信号抑制静音物体,提升对发声目标的追踪能力
近期提出的音视频实例分割(AVIS)任务旨在识别、分割并跟踪视频中的发声实例。然而,现有方法多采用离线模式,无法在连续片段间关联检测结果,难以适应真实场景的持续视频流。为此,我们提出SeaVIS,首个面向在线音视频实例分割的框架。SeaVIS通过因果交叉注意力融合(CCAF)模块,将当前帧视觉特征与完整音频历史信息在严格因果约束下融合,实现高效在线处理。传统视觉实例分割方法因依赖外观特征,在区分发声与静音状态时易出错,导致静音物体被错误分割。为此,我们设计了音频引导对比学习(AGCL)策略,生成同时包含视觉外观和发声活动信息的实例原型。该机制可有效抑制未发声实例在关联过程中的留存,显著增强模型的音频跟随能力。在AVISeg数据集上的大量实验表明,SeaVIS在多个评估指标上超越现有最优模型,同时保持适合实时处理的推理速度。
原文摘要 · Abstract (English)
Recently, an audio-visual instance segmentation (AVIS) task has been introduced, aiming to identify, segment and track individual sounding instances in videos. However, prevailing methods primarily adopt the offline paradigm, that cannot associate detected instances across consecutive clips, making them unsuitable for real-world scenarios that involve continuous video streams. To address this limitation, we introduce SeaVIS, the first online framework designed for audio-visual instance segmentation. SeaVIS leverages the Causal Cross Attention Fusion (CCAF) module to enable efficient online processing, which integrates visual features from the current frame with the entire audio history under strict causal constraints. A major challenge for conventional VIS methods is that appearance-based instance association fails to distinguish between an object's sounding and silent states, resulting in the incorrect segmentation of silent objects. To tackle this, we employ an Audio-Guided Contrastive Learning (AGCL) strategy to generate instance prototypes that encode not only visual appearance but also sounding activity. In this way, instances preserved during per-frame prediction that do not emit sound can be effectively suppressed during instance association process, thereby significantly enhancing the audio-following capability of SeaVIS. Extensive experiments conducted on the AVISeg dataset demonstrate that SeaVIS surpasses existing state-of-the-art models across multiple evaluation metrics while maintaining a competitive inference speed suitable for real-time processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。