针对真实遥感图像,实现快速自适应的高精度融合。
CAT: A Conditional Adaptation Tailor for Efficient and Effective Instance-Specific Pansharpening on Real-World Data
- 输入图像分块后,仅用部分块无监督训练,快速适配当前图像。
- 512×512图0.4秒内完成训练与推理,4000×4000图3秒内完成。
- 无需重训即可跨传感器提升效果,适合实时遥感应用。
pansharpening 是将低分辨率多光谱(LRMS)图像与高分辨率全色(PAN)图像融合生成高分辨率多光谱(HRMS)影像的关键遥感技术。尽管深度学习显著提升了该任务性能,但多数方法存在跨传感器泛化能力差、计算开销大等问题,限制了其在实际场景中的实时应用。为此,本文提出一种高效框架,可快速适配特定输入实例,在短时间内完成训练与推理。框架将输入图像划分为多个块,选取子集进行无监督的条件自适应调制(CAT)训练,随后对所有块进行推理并拼接输出。CAT模块嵌入预训练网络的特征提取与通道变换之间,动态调整融合特征并固定参数以实现高效推理。本方法具备两大优势:(1) 改进的跨传感器泛化能力:即使预训练于特定数据集,仍能在其他传感器采集的数据上取得更优表现;(2) 提升的计算效率:仅需单个LRMS-PAN图像对即可快速适配测试样本,无需大规模重训练。在WorldView-3和WorldView-2的真实数据上的实验表明,本方法在跨传感器真实数据上达到领先性能,且512×512图像的训练与推理耗时不超过0.4秒,4000×4000图像在最快设置下不超过3秒(基于RTX 3090 GPU)。
原文摘要 · Abstract (English)
Pansharpening is a crucial remote sensing technique that fuses low-resolution multispectral (LRMS) images with high-resolution panchromatic (PAN) images to generate high-resolution multispectral (HRMS) imagery. Although deep learning techniques have significantly advanced pansharpening, many existing methods suffer from limited cross-sensor generalization and high computational overhead, restricting their real-time applications. To address these challenges, we propose an efficient framework that quickly adapts to a specific input instance, completing both training and inference in a short time. Our framework splits the input image into multiple patches, selects a subset for unsupervised CAT training, and then performs inference on all patches, stitching them into the final output. The CAT module, integrated between the feature extraction and channel transformation stages of a pre-trained network, tailors the fused features and fixes the parameters for efficient inference, generating improved results. Our approach offers two key advantages: (1) $\textit{Improved Generalization Ability}$: by mitigating cross-sensor degradation, our model--although pre-trained on a specific dataset--achieves superior performance on datasets captured by other sensors; (2) $\textit{Enhanced Computational Efficiency}$: the CAT-enhanced network can swiftly adapt to the test sample using the single LRMS-PAN pair input, without requiring extensive large-scale data retraining. Experiments on the real-world data from WorldView-3 and WorldView-2 datasets demonstrate that our method achieves state-of-the-art performance on cross-sensor real-world data, while achieving both training and inference of $512\times512$ image within $\textit{0.4 seconds}$ and $4000\times4000$ image within $\textit{3 seconds}$ at the fastest setting on a commonly used RTX 3090 GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。