无需标注数据，融合图像与事件流实现更准的单目深度估计
FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion
- 通过频率解耦对齐与自监督迁移，解决跨模态数据少和频率不匹配问题
- 在MVSEC和DENSE数据集上分别提升14%和24.9%的绝对相对误差表现
- 零样本适配极端光照与运动模糊场景，适合真实世界部署
图像-事件联合深度估计方法利用互补模态提升感知鲁棒性，但面临两大挑战：一是标注图像-事件-深度数据集稀缺导致跨模态监督不足；二是静态图像与动态事件流在时空模式上存在固有频率差异，造成特征融合效率低下。为此，我们提出频率解耦统一自监督编码器（FUSE），包含两个协同组件：参数高效的自监督迁移（PST）通过图像基础模型的潜在空间对齐实现跨模态知识迁移，在无深度真值条件下完成联合编码，缓解数据稀缺问题；同时提出频率解耦融合模块（FreDFuse），显式分离高频边缘特征与低频结构成分，通过物理感知融合解决模态间频率失配。该框架可构建通用图像-事件编码器，仅需轻量解码器适配即可用于目标数据集。大量实验表明，其在MVSEC和DENSE数据集上绝对相对误差（Abs .Rel）分别提升14%和24.9%。方法在极端光照与运动模糊等挑战性场景下展现显著零样本适应能力，大幅增强实际部署潜力。代码已开源：https://github.com/sunpihai-up/FUSE
原文摘要 · Abstract (English)
Image-event joint depth estimation methods leverage complementary modalities for robust perception, yet face challenges in generalizability stemming from two factors: 1) limited annotated image-event-depth datasets causing insufficient cross-modal supervision, and 2) inherent frequency mismatches between static images and dynamic event streams with distinct spatiotemporal patterns, leading to ineffective feature fusion. To address this dual challenge, we propose Frequency-decoupled Unified Self-supervised Encoder (FUSE) with two synergistic components: The Parameter-efficient Self-supervised Transfer (PST) establishes cross-modal knowledge transfer through latent space alignment with image foundation models, effectively mitigating data scarcity by enabling joint encoding without depth ground truth. Complementing this, we propose the Frequency-Decoupled Fusion module (FreDFuse) to explicitly decouple high-frequency edge features from low-frequency structural components, resolving modality-specific frequency mismatches through physics-aware fusion. This combined approach enables FUSE to construct a universal image-event encoder that only requires lightweight decoder adaptation for target datasets. Extensive experiments demonstrate state-of-the-art performance with 14% and 24.9% improvements in Abs .Rel on MVSEC and DENSE datasets. The framework exhibits remarkable zero-shot adaptability to challenging scenarios including extreme lighting and motion blur, significantly advancing real-world deployment capabilities. The source code for our method is publicly available at: https://github.com/sunpihai-up/FUSE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。