不依赖特定预训练，用波段选择让多模态模型适配高光谱任务
Spectral Gaps and Spatial Priors: Studying Hyperspectral Downstream Adaptation Using TerraMind
- 通过波段选择和物理响应函数分组两种策略适配高光谱数据
- 模型经波段选择后仍可完成下游任务，性能略有下降
- 揭示未来模型需原生支持光谱编码，为高光谱融合提供基准
地理空间基础模型（GFMs）通常缺乏对高维光谱数据的原生支持。本研究探讨了TerraMind这一多模态基础模型在未进行高光谱特定预训练的情况下，对高光谱成像（HSI）下游任务的适应能力。为此，我们实施并对比了两种通道适配策略：朴素波段选择与基于物理先验的光谱响应函数（SRF）分组。实验表明，具有原生高光谱支持的深度学习模型整体表现更优。同时，研究发现通过波段选择，TerraMind可有效适配高光谱下游任务，性能仅出现中等程度下降。结果为高光谱集成建立了关键基线，强调未来多模态模型架构需具备原生光谱标记化能力。
原文摘要 · Abstract (English)
Geospatial Foundation Models (GFMs) typically lack native support for Hyperspectral Imaging (HSI) due to the complexity and sheer size of high-dimensional spectral data. This study investigates the adaptability of TerraMind, a multimodal GFM, to address HSI downstream tasks \emph{without} HSI-specific pretraining. Therefore, we implement and compare two channel adaptation strategies: Naive Band Selection and physics-aware Spectral Response Function (SRF) grouping. Overall, our results indicate a general superiority of deep learning models with native support of HSI data. Our experiments also demonstrate the ability of TerraMind to adapt to HSI downstream tasks through band selection with moderate performance decline. Therefore, the findings of this research establish a critical baseline for HSI integration, motivating the need for native spectral tokenization in future multimodal model architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。