首个多模态视频融合框架,兼顾时序一致性与语义视觉协同。
TemCoCo: Temporally Consistent Multi-modal Video Fusion with Visual-Semantic Collaboration
- 引入视觉-语义交互模块,用Dinov2和VGG19分别优化双分支表示。
- 构建时序协作模块,利用时间依赖性恢复弱信息,提升融合质量。
- 设计时序损失与新评估指标,专门衡量生成视频的时序一致性。
现有多模态融合方法通常直接套用静态图像融合技术于视频任务,忽略固有时序依赖,导致帧间结果不一致。为此,我们提出首个显式结合时序建模与视觉-语义协同的视频融合框架,同时保障视觉保真度、语义准确性和时序一致性。首先,设计视觉-语义交互模块,包含语义分支与视觉分支,分别采用Dinov2和VGG19进行针对性知识蒸馏,实现双表示同步增强。其次,首次将视频退化增强任务引入融合流程,构建时序协作模块,利用时间依赖性促进弱信息恢复。第三,嵌入时序增强机制并设计时序损失,指导网络优化以确保时序一致性。最后,提出两项专为视频融合设计的新评估指标,用于量化生成视频的时序一致性。在多个公开视频数据集上的实验表明,本方法显著优于现有方法。代码已开源:https://github.com/Meiqi-Gong/TemCoCo。
原文摘要 · Abstract (English)
Existing multi-modal fusion methods typically apply static frame-based image fusion techniques directly to video fusion tasks, neglecting inherent temporal dependencies and leading to inconsistent results across frames. To address this limitation, we propose the first video fusion framework that explicitly incorporates temporal modeling with visual-semantic collaboration to simultaneously ensure visual fidelity, semantic accuracy, and temporal consistency. First, we introduce a visual-semantic interaction module consisting of a semantic branch and a visual branch, with Dinov2 and VGG19 employed for targeted distillation, allowing simultaneous enhancement of both the visual and semantic representations. Second, we pioneer integrate the video degradation enhancement task into the video fusion pipeline by constructing a temporal cooperative module, which leverages temporal dependencies to facilitate weak information recovery. Third, to ensure temporal consistency, we embed a temporal-enhanced mechanism into the network and devise a temporal loss to guide the optimization process. Finally, we introduce two innovative evaluation metrics tailored for video fusion, aimed at assessing the temporal consistency of the generated fused videos. Extensive experimental results on public video datasets demonstrate the superiority of our method. Our code is released at https://github.com/Meiqi-Gong/TemCoCo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。