用RGB图像实现多光谱感知，提升卫星图像理解能力
Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery
- 通过轻量投影将多光谱知识蒸馏到仅用RGB的模型
- 在三个数据集上零样本分类提升4.2%，检索提升5.9%
- 适合需要低成本部署的遥感视觉语言任务
视觉-语言基础模型（VLFMs）有望实现地球观测的零样本与检索理解。尽管实际卫星系统常缺乏全多光谱覆盖，仅使用RGB输入更利于规模化部署，但当前VLFMs在卫星图像中的应用仍受两大因素制约：(1) 多光谱输入虽具信息量，但波段冗余与错位导致难以一致利用；(2) 类CLIP的文本编码器限制语义表达力，削弱细粒度对齐。本文提出SATtxt，一种仅在推理时使用RGB输入，但训练中保留光谱线索的谱感知VLFM。框架分两阶段：首先，通过轻量投影器将冻结的多光谱教师模型中的光谱先验蒸馏至RGB学生模型；其次，结合指令增强的大语言模型进行谱基对齐，打通蒸馏视觉空间与高表达力语言空间。在EuroSAT、BigEarthNet和ForestNet上，SATtxt平均实现零样本分类提升4.2%、检索性能提升5.9%、线性探测提升2.7%，为地球观测中高效谱感知视觉-语言学习提供可行路径。
原文摘要 · Abstract (English)
Vision-language foundation models (VLFMs) promise zero-shot and retrieval understanding for Earth observation. While operational satellite systems often lack full multi-spectral coverage, making RGB-only inference highly desirable for scalable deployment, the adoption of VLFMs for satellite imagery remains hindered by two factors: (1) multi-spectral inputs are informative but difficult to exploit consistently due to band redundancy and misalignment; and (2) CLIP-style text encoders limit semantic expressiveness and weaken fine-grained alignment. We present SATtxt, a spectrum-aware VLFM that operates with RGB inputs only at inference while retaining spectral cues learned during training. Our framework comprises two stages. First, Spectral Representation Distillation transfers spectral priors from a frozen multi-spectral teacher to an RGB student via a lightweight projector. Second, Spectrally Grounded Alignment with Instruction-Augmented LLMs bridges the distilled visual space and an expressive LLM embedding space. Across EuroSAT, BigEarthNet, and ForestNet, SATtxt improves zero-shot classification on average by 4.2%, retrieval by 5.9%, and linear probing by 2.7% over baselines, showing an efficient path toward spectrum-aware vision-language learning for Earth observation. Project page: https://ikhado.github.io/sattxt/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。