用物理规律组合提升模型微调精度,轻量高效。
Mixture of Physical Priors Adapter for Parameter-Efficient Fine-Tuning
- 融合热扩散、波动与泊松方程构建权重先验
- 在VTAB-1K上提升准确率最高2.1%,参数量相当
- 适合需要高精度微调的视觉任务开发者
多数参数高效微调(PEFT)方法依赖低秩表示,但此类方法常过度简化数据,尤其在高秩或高频数据下表现受限。本文提出混合物理先验适配器(MoPPA),通过热扩散、波传播和泊松稳态方程三类物理规律建模网络权重,分别实现局部平滑性、长程交互与全局平衡。采用离散余弦变换(DCT)实现高效计算,并引入路径正则化机制动态调节各先验贡献。MoPPA为轻量级即插即用模块,可融入Transformer架构,复杂度自适应。基于MAE预训练ViT-B,在VTAB-1K图像分类任务中,准确率最高提升2.1%,且在多种视觉主干网络上验证了有效性与泛化能力。代码将公开。
原文摘要 · Abstract (English)
Most parameter-efficient fine-tuning (PEFT) methods rely on low-rank representations to adapt models. However, these approaches often oversimplify representations, particularly when the underlying data has high-rank or high-frequency components. This limitation hinders the model's ability to capture complex data interactions effectively. In this paper, we propose a novel approach that models network weights by leveraging a combination of physical priors, enabling more accurate approximations. We use three foundational equations -- heat diffusion, wave propagation, and Poisson's steady-state equation -- each contributing distinctive modeling properties: heat diffusion enforces local smoothness, wave propagation facilitates long-range interactions, and Poisson's equation captures global equilibrium. To combine these priors effectively, we introduce the Mixture of Physical Priors Adapter (MoPPA), using an efficient Discrete Cosine Transform (DCT) implementation. To dynamically balance these priors, a route regularization mechanism is designed to adaptively tune their contributions. MoPPA serves as a lightweight, plug-and-play module that seamlessly integrates into transformer architectures, with adaptable complexity depending on the local context. Specifically, using MAE pre-trained ViT-B, MoPPA improves PEFT accuracy by up to 2.1% on VTAB-1K image classification with a comparable number of trainable parameters, and advantages are further validated through experiments across various vision backbones, showcasing MoPPA's effectiveness and adaptability. The code will be made public available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。