统一模型同时处理三种马赛克模式,提升手机多摄像头图像质量
Examining Joint Demosaicing and Denoising for Single-, Quad-, and Nona-Bayer Patterns
- 用嵌入向量统一处理单、四、九宫格马赛克图案
- 性能超越现有四、九宫格模型,接近单通道模型水平
- 新增掩码策略支持坏点修复,适合多摄像头系统开发
相机传感器采用马赛克布局的色滤阵列,传统为Bayer模式。去马赛克是获取全通道RGB图像的关键步骤。如今许多智能手机配备多种不同布局的传感器,如Quad-Bayer或Nona-Bayer。现有基于深度网络的模型通常为每种模式训练独立网络,导致内存开销大且切换困难。本文研究三种主要马赛克布局(1x1 Single-Bayer、2x2 Quad-Bayer、3x3 Nona-Bayer)的联合去马赛克与去噪策略。通过在输入图像中拼接三通道马赛克嵌入,并训练统一架构,结果优于现有Quad-Bayer和Nona-Bayer模型,且接近Single-Bayer模型表现。此外,提出一种掩码策略,提升模型性能并实现坏点修正——该功能常被现有AI去马赛克模型忽略。为此,我们采集了包含638张RAW图像的新数据集,涵盖复杂场景,并对图像块进行标注以供训练、验证与测试。代码与数据已公开于https://github.com/SamsungLabs/unified-demosaicing。
原文摘要 · Abstract (English)
Camera sensors have color filters arranged in a mosaic layout, traditionally following the Bayer pattern. Demosaicing is a critical step camera hardware applies to obtain a full-channel RGB image. Many smartphones now have multiple sensors with different patterns, such as Quad-Bayer or Nona-Bayer. Most modern deep network-based models perform joint demosaicing and denoising with the strategy of training a separate network per pattern. Relying on individual models per pattern requires additional memory overhead and makes it challenging to switch quickly between cameras. In this work, we are interested in analyzing strategies for joint demosaicing and denoising for the three main mosaic layouts (1x1 Single-Bayer, 2x2 Quad-Bayer, and 3x3 Nona-Bayer). We found concatenating a three-channel mosaic embedding to the input image and training a unified demosaicing architecture yields results that outperform existing Quad-Bayer and Nona-Bayer models and are comparable to Single-Bayer models. Additionally, we describe a maskout strategy that enhances the model performance and facilitates dead pixel correction -- a step often overlooked by existing AI-based demosaicing models. As part of this effort, we captured a new demosaicing dataset of 638 RAW images that contain challenging scenes with patches annotated for training, validation, and testing. Code and data is available at https://github.com/SamsungLabs/unified-demosaicing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。