遥感大模型冗余度高,剪枝后仍保持高精度,适合资源受限场景。
How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models
- 通过统一缩窄编码器结构,测试模型冗余性
- 剪掉部分参数后遥感模型仍保持69%~109%准确率
- 适用于遥感任务部署与模型压缩研究
遥感领域的大规模基础模型(RS FMs)沿用计算机视觉的构建范式,但其是否适用自然图像的缩放规律尚不明确。我们假设:相较于计算机视觉模型,遥感模型在更小规模即进入过参数化状态,且任务信息在模型维度间存在冗余编码。为此,我们在分类、分割和变化检测任务上,对八种先进遥感基础模型采用事后剪枝(post-hoc slimmability)方法,进行均匀宽度缩减。结果显示,这些模型在极端宽度缩减下,仍在遥感数据集上保持69%至109%的相对准确率;而基于自然图像预训练的CV MAE与CV DINOv2在相同计算开销下,于匹配类别数的ImageNet子集上表现急剧下降。直接在遥感数据集上评估的CV MAE虽缩小差距,但未完全弥合,说明数据特性与领域预训练共同导致差异。机制分析表明,关键任务方差集中于少数主成分,且在模型维度中被冗余编码。此外,针对对比学习目标,可学习剪枝训练优于事后剪枝,但重建类目标无改善。研究确立事后剪枝为遥感资源受限应用的实用策略,也是探测模型冗余的有效诊断工具。论文接受后将开源全部代码。
原文摘要 · Abstract (English)
Large-scale foundation models (FMs) in remote sensing (RS) (denoted as RS FMs) are developed following paradigms established in computer vision (CV), yet the validity of transferring CV scaling laws to RS has not been systematically examined. We hypothesize that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions. To test this hypothesis, we apply post-hoc slimmability, uniform width reduction of pretrained encoder transformer blocks, as a tool to measure representational redundancy across eight state-of-the-art RS FMs on classification, segmentation, and change detection tasks. RS FMs retain 69% to 109% relative accuracy on RS datasets under aggressive width reduction, while masked autoencoder (MAE) and DINOv2 pretrained on natural images (denoted as CV MAE and CV DINOv2) degrade sharply on ImageNet subsets of matched class count over the same range of computational requirements. A CV MAE evaluated directly on the same RS datasets narrows but does not close the gap, indicating that both dataset characteristics and domain-specific pretraining contribute to the differences between the models. Mechanistic analyses such as feature correlation, explained variance, and effective dimensionality indicate that task-relevant variance concentrates in few principal components and is redundantly encoded across model dimensions. We further show that learned slimmable training improves over post-hoc slimmability for contrastive objectives, while reconstruction-based objectives do not benefit from current slimmable training protocols. Our findings establish post-hoc slimming as a practical deployment strategy for resource-constrained RS applications and as a diagnostic tool for representational redundancy in RS FMs. Upon acceptance, we will publish all code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。