用高效微调让多光谱大模型搞定高光谱分类,节省99%参数和存储。
Parameter-Efficient Fine-Tuning of Multispectral Foundation Models for Hyperspectral Image Classification
- 采用改进的KronA+方法,仅0.056%参数可训练
- 在5个数据集上性能接近专用模型,训练只需四分之一周期
- 适合资源有限但需高光谱分类的遥感应用
基础模型在遥感领域取得显著进展,但多数针对多光谱数据,而拥有数百个波段的高光谱影像(HSI)仍较少被探索。将此类模型微调用于下游任务通常需要大量内存与存储。本文提出一种高效框架,对多光谱基础模型SpectralGPT进行高光谱图像分类(HSIC)微调。评估了多种参数高效微调(PEFT)方法,包括低秩适配(LoRA)、基于克罗内克积的适配(KronA)、低秩克罗内克(LoKr)以及近期的LoRA+(使用不同学习率缩放低秩适配器)。受LoRA+启发,引入KronA+,对克罗内克矩阵应用类似机制。在五个来自不同传感器的数据集上验证,性能媲美当前最优的高光谱模型。全量微调(FFT)设置下,SpectralGPT甚至在部分数据集上超越专用高光谱基础模型,且训练周期仅为四分之一。在相同训练轮次下,KronA+仅需0.056%的可训练参数,额外存储约0.2兆字节,是测试中最高效的PEFT方法。
原文摘要 · Abstract (English)
Foundation models have achieved great success across diverse domains, including remote sensing (RS), thanks to their versatility and strong generalization abilities. However, most RS foundation models are designed for multispectral data, while hyperspectral imagery (HSI) - with its hundreds of spectral bands - remains less explored. Fine-tuning such models for downstream tasks is also challenging, often demanding considerable memory and storage. In this paper, we propose an efficient framework to fine-tune SpectralGPT, a multispectral foundation model, for hyperspectral image classification (HSIC). We explore several Parameter-Efficient Fine-Tuning (PEFT) methods, including Low-Rank Adaptation (LoRA), Kronecker-based adaptation (KronA), Low-Rank Kronecker (LoKr), and the recent LoRA+, which uses distinct learning rates for low-rank adapters scaled by a factor lambda. Inspired by LoRA+, we introduce KronA+, which applies a similar mechanism to the Kronecker matrices. We evaluate our approach on five datasets from different sensors, showing competitive performance with state-of-the-art HSI models. Our full fine-tuning (FFT) setup for SpectralGPT even outperforms a dedicated hyperspectral foundation model on some datasets while requiring only a quarter of the training epochs. Under the same number of epochs, KronA+ reaches similar performance with far fewer trainable parameters - just 0.056 percent - and adds only approximately 0.2 megabytes of storage, making it the most effective PEFT method tested.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。