arXiv:2412.08161cs.CVcs.LG2024-12被引 1

解决音视频分割中的时间错位问题,提升声音与画面的同步精度。

Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation

  • 通过语义边界锚定和逐帧音频插入传播,精准捕捉声音起止点。
  • 在三个数据集上实现更优对齐率,内存消耗更低且支持即插即用。
  • 适合需要高时序精度的音视频理解任务,如智能剪辑、交互式视频分析。

音视频视频分割(AVVS)旨在生成与音频精确对齐的声音源物体像素级掩码。然而现有方法常面临时间错位问题,即音频线索与分割结果未在时间上协调。音频提供两类关键信息:一是目标物体层面的细节,二是物体发声起止的时间。当前方法更关注物体层面信息,忽视了音频语义变化的边界,导致时间错位。为此,我们提出协同混合传播框架(Co-Prop),包含两步:初步音频边界锚定与逐帧音频插入传播。通过Qwen大模型辅助检索提示识别音频语义变化的控制点,将音频划分为语义一致段落。随后,采用逐帧音频插入传播与匹配方法处理各段落。我们构建了一个涵盖多种音源转换场景的小型数据集,并设计评估指标衡量对齐率。相比传统同步处理方法,本方法降低内存开销并支持帧级对齐。实验表明,在三个数据集和两种主干网络上均有效。此外,该方法可无缝集成至现有AVVS模型,实现即插即用性能提升。

原文摘要 · Abstract (English)

Audio-visual video segmentation (AVVS) aims to generate pixel-level maps of sound-producing objects that accurately align with the corresponding audio. However, existing methods often face temporal misalignment, where audio cues and segmentation results are not temporally coordinated. Audio provides two critical pieces of information: i) target object-level details and ii) the timing of when objects start and stop producing sounds. Current methods focus more on object-level information but neglect the boundaries of audio semantic changes, leading to temporal misalignment. To address this issue, we propose a Collaborative Hybrid Propagator Framework~(Co-Prop). This framework includes two main steps: Preliminary Audio Boundary Anchoring and Frame-by-Frame Audio-Insert Propagation. To Anchor the audio boundary, we employ retrieval-assist prompts with Qwen large language models to identify control points of audio semantic changes. These control points split the audio into semantically consistent audio portions. After obtaining the control point lists, we propose the Audio Insertion Propagator to process each audio portion using a frame-by-frame audio insertion propagation and matching approach. We curated a compact dataset comprising diverse source conversion cases and devised a metric to assess alignment rates. Compared to traditional simultaneous processing methods, our approach reduces memory requirements and facilitates frame alignment. Experimental results demonstrate the effectiveness of our approach across three datasets and two backbones. Furthermore, our method can be integrated with existing AVVS approaches, offering plug-and-play functionality to enhance their performance.

音视频分割时间对齐多模态框架设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。