针对气象模型设计高效微调框架,实现小参数量媲美全量微调。
Task-Adaptive Parameter-Efficient Fine-Tuning for Weather Foundation Models
- 动态注入任务相关提示,自适应调整特征表示。
- 结合费雪信息与随机性,精准选择关键可训练参数。
- 在3个气象任务上以极少参数达全微调效果,适合资源受限场景。
尽管近期机器学习进展使气象基础模型(WFMs)具备跨多种下游任务的强大泛化能力,但其规模扩大带来的计算需求日益阻碍实际部署。现有参数高效微调(PEFT)方法针对视觉或语言任务设计,难以应对气象任务特有的异质性、分辨率多样性和时空覆盖差异,导致在WFMs上性能不佳。为此,我们提出WeatherPEFT,一种融合两项协同创新的新型PEFT框架:首先,在前向传播中,任务自适应动态提示(TADP)通过内部与外部模式提取,将编码器中的嵌入权重动态注入预训练主干的输入令牌,实现特定任务的上下文感知特征重校准;其次,在反向传播中,随机费雪引导自适应选择(SFAS)利用费雪信息识别并更新最任务敏感的参数,同时引入随机性以稳定选择过程,从而保留预训练知识不变性。我们在三个下游任务上验证了WeatherPEFT的有效性与效率,相比全微调,现有PEFT方法存在显著差距,而WeatherPEFT仅用更少可训练参数即达到全微调性能。代码已开源:https://github.com/ShileiCao/WeatherPEFT。
原文摘要 · Abstract (English)
While recent advances in machine learning have equipped Weather Foundation Models (WFMs) with substantial generalization capabilities across diverse downstream tasks, the escalating computational requirements associated with their expanding scale increasingly hinder practical deployment. Current Parameter-Efficient Fine-Tuning (PEFT) methods, designed for vision or language tasks, fail to address the unique challenges of weather downstream tasks, such as variable heterogeneity, resolution diversity, and spatiotemporal coverage variations, leading to suboptimal performance when applied to WFMs. To bridge this gap, we introduce WeatherPEFT, a novel PEFT framework for WFMs incorporating two synergistic innovations. First, during the forward pass, Task-Adaptive Dynamic Prompting (TADP) dynamically injects the embedding weights within the encoder to the input tokens of the pre-trained backbone via internal and external pattern extraction, enabling context-aware feature recalibration for specific downstream tasks. Furthermore, during backpropagation, Stochastic Fisher-Guided Adaptive Selection (SFAS) not only leverages Fisher information to identify and update the most task-critical parameters, thereby preserving invariant pre-trained knowledge, but also introduces randomness to stabilize the selection. We demonstrate the effectiveness and efficiency of WeatherPEFT on three downstream tasks, where existing PEFT methods show significant gaps versus Full-Tuning, and WeatherPEFT achieves performance parity with Full-Tuning using fewer trainable parameters. The code of this work is available at https://github.com/ShileiCao/WeatherPEFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。