用生成模型辅助训练,让多光谱相机去马赛克更准、跨设备通用。
Generative Model-Assisted Demosaicing for Cross-multispectral Cameras
- 用生成模型自动生成真实数据的伪标签,解决标注难问题。
- 在真实数据上跨相机测试,性能优于现有方法2.5~4.3dB。
- 识别易出错区域并针对性优化,减少图像伪影,适合工程部署。
基于光谱滤波阵列的多光谱成像中,光谱去马赛克因深度学习兴起而快速发展。然而,(1) 实际数据难以获取对应标签或模拟真实成像过程,导致基于模拟数据训练的端到端网络在真实数据上表现差;(2) 不同相机间的光谱差异使预训练模型难以迁移;(3) 现有网络在复杂场景中插值未知值易引入视觉伪影。为此,我们提出一种结合自监督生成模型的混合监督训练方法,在不同光谱相机的真实数据上均表现良好。具体分三步:(1) 在大量模拟数据上预训练端到端神经网络;(2) 使用自监督生成模型生成目标真实数据的伪标签;(3) 在第二步生成的伪数据对上微调预训练模型。为缓解伪影,提出频域硬块选择方法,通过傅里叶变换与滤波分析光谱差异,定位易出错区域,实现针对性微调。最后,构建了真实世界多光谱马赛克图像测试集UniSpecTest。消融实验证明各步骤有效,合成与真实数据上的大量实验表明,本方法相比最先进技术显著提升性能,平均提升2.5~4.3dB。
原文摘要 · Abstract (English)
As a crucial part of the spectral filter array (SFA)-based multispectral imaging process, spectral demosaicing has exploded with the proliferation of deep learning techniques. However, (1) bothering by the difficulty of capturing corresponding labels for real data or simulating the practical spectral imaging process, end-to-end networks trained in a supervised manner using simulated data often perform poorly on real data. (2) cross-camera spectral discrepancies make it difficult to apply pre-trained models to new cameras. (3) existing demosaicing networks are prone to introducing visual artifacts on hard cases due to the interpolation of unknown values. To address these issues, we propose a hybrid supervised training method with the assistance of the self-supervised generative model, which performs well on real data across different spectral cameras. Specifically, our approach consists of three steps: (1) Pre-Training step: training the end-to-end neural network on a large amount of simulated data; (2) Pseudo-Pairing step: generating pseudo-labels of real target data using the self-supervised generative model; (3) Fine-Tuning step: fine-tuning the pre-trained model on the pseudo data pairs obtained in (2). To alleviate artifacts, we propose a frequency-domain hard patch selection method that identifies artifact-prone regions by analyzing spectral discrepancies using Fourier transform and filtering techniques, allowing targeted fine-tuning to enhance demosaicing performance. Finally, we propose UniSpecTest, a real-world multispectral mosaic image dataset for testing. Ablation experiments have demonstrated the effectiveness of each training step, and extensive experiments on both synthetic and real datasets show that our method achieves significant performance gains compared to state-of-the-art techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。