用冻结的CLIP特征做图像重建，自监督下效果接近有标签训练。
Frozen CLIP Priors for Robust Self-Supervised Poisson Inverse Problems
- 用ADMM思路解耦数据一致性与轻量级先验，先验基于冻结的CLIP特征。
- 在泊松噪声下表现稳定，自监督性能接近有监督，支持多种成像变换。
- 适合低光、无真值数据的图像重建，尤其适合多设备/采集条件变化场景。
在光子受限场景中，获取干净真值数据不现实，重建需在数据集和采集方式变化下保持稳定。这一挑战在泊松噪声下更严重，因其信号依赖性统计与采样算子（如色彩滤波阵列马赛克）相互作用。大规模预训练视觉编码器（如CLIP RN50）提供不变于失真的内容表征，具有跨域泛化能力，是构建无需昂贵微调即可迁移的先验的理想路径。本文提出一种受ADMM启发的可展开式插件式求解器，将闭式数据一致性更新与参数高效的先验分离。先验采用轻量级解码器，基于冻结的CLIP RN50密集多尺度特征进行适应。为实现自监督，方法结合测量域重污染（GR2R）与等变成像正则化，通过虚拟采集实现。在泊松型色彩滤波阵列去马赛克与去模糊任务上，该方法达到竞争性质量，对分布外变化具有更强鲁棒性，自监督性能逼近有监督训练。
原文摘要 · Abstract (English)
Self-supervised learning for imaging inverse problems is increasingly important in photon-limited settings, where acquiring clean ground truth is impractical and reconstruction must remain stable under dataset and acquisition shifts. This challenge is amplified under Poisson noise, whose signal-dependent statistics interact with sampling operators (e.g., CFA mosaicing). Meanwhile, foundation vision encoders trained at web scale offer distortion-invariant, content-related representations that generalize well across domains, suggesting a promising route to build priors that transfer beyond the training distribution without expensive fine-tuning. This paper proposes an ADMM-inspired unrolled plug-and-play solver for Poisson inverse problems that decouples a closed-form data-consistency update from a parameter-efficient prior. The prior is implemented as a lightweight decoder operating on frozen CLIP RN50 dense multi-scale features, adapting foundation representations with less trainable parameters. For self-supervision, the method integrates GR2R measurement-domain re-corruption with an Equivariant Imaging regularizer via virtual acquisitions. Experiments on Poisson CFA demosaicing and deblurring show competitive quality, improved robustness under shifts, and self-supervised performance approaching supervised training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。