用分割信息提升图像去马赛克效果,让不同任务共享结构理解。
SegDem: Segmentation helps Demosaicing

- 通过实例分割预训练获取区域与边界感知特征
- 在多种传感器布局下均实现显著性能提升
- 适合需要高质量色彩重建的图像处理研究者
图像去马赛克是从被色滤阵列(CFA)遮挡的不完整颜色测量中恢复全彩图像的过程。现有方法多将去马赛克视为像素级重建,主要依赖局部纹理、跨通道相关性和低层图像统计。本文的核心洞察是:重建与视觉理解可视为共享场景结构的互补视角——二者均基于同一物理世界,因此从图像中推断出的结构与物理信息应在两任务间保持一致。为此,我们提出 extit{SegDem},一种基于实例分割的跨任务解码器表征迁移框架。SegDem 首先通过实例感知结构预训练学习区域与边界感知表征,再将解码器迁移到 RAW 条件下的重建任务。分割与去马赛克条件特征进一步锚定至共享的冻结 DINOv2 表征空间,以保持任务间的结构一致性。我们在卷积、Transformer 和状态空间模型三种骨干网络上实现了统一的单像素与四像素马赛克去马赛克。在合成、外部及挑战性数据集上的大量实验表明,该方法在不同架构和 CFA 布局下均取得持续提升。
原文摘要 · Abstract (English)
Image demosaicing reconstructs a full-color image from incomplete color measurements produced by a sensor covered with a color filter array (CFA). Most existing methods formulate demosaicing as pixel-level reconstruction and mainly rely on local textures, cross-channel correlations, and low-level image statistics. Our core insight is that reconstruction and visual understanding can be viewed as complementary views of shared scene structure: both are grounded in the same underlying physical world, and therefore the structural and physical information inferred from an image should remain consistent across the two tasks. We instantiate this idea with instance segmentation and propose \emph{SegDem}, a cross-task decoder representation transfer framework for demosaicing. SegDem first learns region- and boundary-aware representations through instance-aware structural pretraining and then transfers the decoder to RAW-conditioned reconstruction. Segmentation- and demosaicing-conditioned features are further anchored to a shared frozen DINOv2 representation space to preserve structural organization across tasks. We instantiate SegDem with convolutional, Transformer-based, and state-space backbones for unified Single- and Quad-Bayer demosaicing. Extensive experiments on synthetic, external, and challenging datasets demonstrate consistent improvements across different architectures and CFA layouts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。