针对科学计算中的大模型,提出频域自适应微调方法F-Adapter
F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning
- 根据频谱复杂度分配参数,低频用高维模块,高频用低维模块
- 在3D纳维-斯托克斯方程上超越LoRA等方法,提升泛化与频谱保真度
- 首次系统探索科学机器学习的参数高效微调,适合物理建模研究者
将参数高效微调(PEFT)应用于预训练的大算子模型(LOMs),这些模型基于傅里叶神经算子的变体构建。我们发现,广泛使用的低秩适配(LoRA)在LOMs上的表现显著差于适配器(Adapter)调优。理论上证明:堆叠的LoRA在傅里叶层中引入深度放大的近似误差下界,而适配器保持通用逼近能力,并通过集中参数于能量主导的低频模式,在傅里叶域实现随瓶颈宽度指数衰减的误差。鉴于适配器的实证优势及偏微分方程解的频谱稀疏特性,我们提出频域自适应适配器(F-Adapter),按频谱复杂度分配容量:低频用高维模块,高频用低维模块。F-Adapter在多个挑战性的3D纳维-斯托克斯基准测试中达到最先进水平,显著提升泛化性能和频谱保真度,优于LoRA及其他常用于大语言模型的PEFT技术。据我们所知,这是首个探索科学机器学习中PEFT的工作,确立了F-Adapter作为该领域的有效范式。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) of powerful pre-trained models for complex downstream tasks has proven effective in vision and language processing, yet this paradigm remains unexplored in scientific machine learning, where the objective is to model complex physical systems. We conduct the first systematic study of PEFT for pre-trained Large Operator Models (LOMs) obtained by scaling variants of Fourier Neural Operator. First, we observe that the widely used Low-Rank Adaptation (LoRA) yields markedly poorer performance on LOMs than Adapter tuning. Then, we further theoretically establish that stacked LoRA incurs a depth-amplified lower bound on approximation error within Fourier layers, whereas adapters retain universal approximation capacity and, by concentrating parameters on energy-dominant low-frequency modes, attain exponentially decaying error with bottleneck width in the Fourier domain. Motivated by the robust empirical gains of adapters and by our theoretical characterization of PDE solutions as spectrally sparse, we introduce Frequency-Adaptive Adapter (F-Adapter). F-Adapter allocates adapter capacity based on spectral complexity, assigning higher-dimension modules to low-frequency components and lower-dimension modules to high-frequency components. Our F-Adapters establish state-of-the-art (SOTA) results on multiple challenging 3D Navier-Stokes benchmarks, markedly enhancing both generalization and spectral fidelity over LoRA and other PEFT techniques commonly used in LLMs. To the best of our knowledge, this work is the first to explore PEFT for scientific machine-learning and establishes F-Adapter as an effective paradigm for this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。