用复数神经网络融合事件数据与图像,提升暗光视频去模糊效果
CompEvent: Complex-valued Event-RGB Fusion for Low-light Video Enhancement and Deblurring
- 通过复数循环单元实现视频与事件流的时序对齐与连续融合
- 在时空域联合处理,显著提升暗光条件下的去模糊性能
- 适合夜间监控、自动驾驶等低光动态场景应用
暗光视频去模糊在夜间监控和自动驾驶中面临巨大挑战,源于光照不足与长曝光导致的模糊。事件相机虽具备优异的低光敏感性和高时间分辨率,但现有融合方法多采用分阶段策略,难以应对光照与运动模糊的双重退化。为此,我们提出CompEvent,一种基于复数神经网络的端到端融合框架,实现事件数据与RGB帧的全过程联合恢复。其核心包含:1)复数时序对齐GRU,利用复数卷积迭代处理视频与事件流,实现时序对齐与连续融合;2)复数空频学习模块,在空间与频率域统一进行复数信号处理,通过结构与系统特性实现深层融合。依托复数网络的全表示能力,CompEvent实现全流程时空融合,最大化模态间互补性,显著增强暗光视频去模糊能力。大量实验表明,CompEvent在该任务上超越现有最先进方法。
原文摘要 · Abstract (English)
Low-light video deblurring poses significant challenges in applications like nighttime surveillance and autonomous driving due to dim lighting and long exposures. While event cameras offer potential solutions with superior low-light sensitivity and high temporal resolution, existing fusion methods typically employ staged strategies, limiting their effectiveness against combined low-light and motion blur degradations. To overcome this, we propose CompEvent, a complex neural network framework enabling holistic full-process fusion of event data and RGB frames for enhanced joint restoration. CompEvent features two core components: 1) Complex Temporal Alignment GRU, which utilizes complex-valued convolutions and processes video and event streams iteratively via GRU to achieve temporal alignment and continuous fusion; and 2) Complex Space-Frequency Learning module, which performs unified complex-valued signal processing in both spatial and frequency domains, facilitating deep fusion through spatial structures and system-level characteristics. By leveraging the holistic representation capability of complex-valued neural networks, CompEvent achieves full-process spatiotemporal fusion, maximizes complementary learning between modalities, and significantly strengthens low-light video deblurring capability. Extensive experiments demonstrate that CompEvent outperforms SOTA methods in addressing this challenging task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。