针对手机传感器的多帧超分，解决大运动下的模糊与伪影问题。
LatentBurst: A Fast and Efficient Multi Frame Super-Resolution for Hexadeca-Bayer Pattern CIS images
- 在特征空间做金字塔对齐融合，应对大运动导致的图像错位。
- 轻量UNet结构支持移动端实时运行，速度优于现有方法。
- 结合知识蒸馏与光学流优化，显著减少领域差异带来的性能损失。
本文提出一种新型多帧超分辨率网络（MFSR），用于处理突发采集的十六色阵列接触式图像传感器（Hexadeca-Bayer Pattern CIS）图像，集成去马赛克、去噪、多帧融合与超分辨率功能。设计高质量重建网络面临三大挑战：1）相比传统拜耳阵列，十六色阵列中同色像素间距增大，插值难度高；2）大物体运动和相机抖动导致多帧对齐困难，融合后易产生模糊或鬼影伪影；3）模型需足够快速高效，以实现在移动设备上的实时运行。为此，我们提出名为LatentBurst的新网络：1）采用特征空间中的金字塔对齐与融合策略，有效处理大运动场景；2）基于轻量级UNet结构，可在移动端高效运行；3）通过微调光流估计与两阶段知识蒸馏，更有效地缩小域间差距。实验结果表明，在多种场景下，该方法显著优于当前主流先进方法。
原文摘要 · Abstract (English)
This paper introduces a novel multi frame super-resolution network (MFSR) for burst hexadeca Bayer pattern Contact Image Sensor (CIS) images, which includes demosaicing, denoising, multi-frame fusion, and super-resolution. Designing a high-quality reconstruction network poses several challenges as follows: 1) Unlike the Bayer color filter array (CFA) pattern, it is hard to interpolate hexadeca-Bayer pattern since the pixel distance between the same color groups increases; 2) Due to large object motion and camera movements, the final fusion result usually suffers the misalignment resulting a blurry image or ghosting artifacts; 3) The proposed network should be fast and efficient enough to operate in real-time on mobile devices. To overcome these challenges, we propose a novel network, called LatentBurst, which contains: 1) a pyramid align and fusion approach in latent feature to deal with large motion scenario; 2) an efficient UNet-based structure which can run efficiently on mobile device; 3) fine-tuned optical flow estimation and two-step knowledge distillation to reduce domain-gap more effectively. Experimental results in various scenarios demonstrate the effectiveness of our proposed method compared with other state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。