提出可动态融合光流与深度信息的新模型,提升RGB-D视频显著性目标检测效果。
Unleashing the Power of Motion and Depth: A Selective Fusion Strategy for RGB-D Video Salient Object Detection
- 根据光流和深度的实际贡献,实现像素级选择性融合
- 在两个数据集上超越19个主流模型,性能领先
- 适合需要多模态视频分析的视觉任务研究者
将显著性目标检测(SOD)应用于RGB-D视频是一项新兴任务,称为RGB-D VSOD,因其结合运动与深度信息可显著提升性能,且日常生活中可轻松获取RGB-D视频。现有模型对运动线索的提取方式各异,其中显式从光流中提取运动信息被认为更有效。然而,如何有效利用光流和深度辅助RGB模态仍是一个关键问题。以往方法在模型设计中对光流与深度同等对待,未考虑其在不同场景下的不均衡贡献,限制了运动与深度的潜力。为此,本文提出一种新型选择性跨模态融合框架SMFNet,包含像素级选择性融合策略(PSF),根据实际贡献最优融合光流与深度。此外,提出多维选择性注意力模块(MSAM),在多个维度上融合PSF输出特征与剩余RGB特征,增强表征能力以生成精炼特征。在RDVS和DVisal两个数据集上对SMFNet与19个先进模型进行全面评估,该评测是当前最全面的RGB-D VSOD基准。结果表明SMFNet优于其他模型。同时,在包含合成深度的五个视频基准数据集上的评估也验证了其有效性。代码与基准结果已公开于https://github.com/Jia-hao999/SMFNet。
原文摘要 · Abstract (English)
Applying salient object detection (SOD) to RGB-D videos is an emerging task called RGB-D VSOD and has recently gained increasing interest, due to considerable performance gains of incorporating motion and depth and that RGB-D videos can be easily captured now in daily life. Existing RGB-D VSOD models have different attempts to derive motion cues, in which extracting motion information explicitly from optical flow appears to be a more effective and promising alternative. Despite this, there remains a key issue that how to effectively utilize optical flow and depth to assist the RGB modality in SOD. Previous methods always treat optical flow and depth equally with respect to model designs, without explicitly considering their unequal contributions in individual scenarios, limiting the potential of motion and depth. To address this issue and unleash the power of motion and depth, we propose a novel selective cross-modal fusion framework (SMFNet) for RGB-D VSOD, incorporating a pixel-level selective fusion strategy (PSF) that achieves optimal fusion of optical flow and depth based on their actual contributions. Besides, we propose a multi-dimensional selective attention module (MSAM) to integrate the fused features derived from PSF with the remaining RGB modality at multiple dimensions, effectively enhancing feature representation to generate refined features. We conduct comprehensive evaluation of SMFNet against 19 state-of-the-art models on both RDVS and DVisal datasets, making the evaluation the most comprehensive RGB-D VSOD benchmark up to date, and it also demonstrates the superiority of SMFNet over other models. Meanwhile, evaluation on five video benchmark datasets incorporating synthetic depth validates the efficacy of SMFNet as well. Our code and benchmark results are made publicly available at https://github.com/Jia-hao999/SMFNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。