arXiv:2505.12532cs.CVcs.AI2025-05被引 6

用小波域稀疏更新,让视觉模型微调更高效精细。

Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Vision

  • 在小波域学习权重矩阵的稀疏更新,突破低秩约束。
  • 视觉任务上性能超越现有方法,语言任务表现相当。
  • 适合需要精细控制参数的视觉模型微调场景。

在计算和内存资源受限的情况下,高效适配大型预训练模型至关重要。尽管参数高效微调(PEFT)方法如LoRA通过低秩更新实现高效,但其离散秩约束限制了参数的细粒度控制,并将适应限制在低维子空间内。本文提出小波微调(WaveFT),在权重矩阵的小波域中学习稀疏更新,实现远低于LoRA最小秩的细粒度参数控制。小波基提供半局部感受野,能聚合空间相干梯度,相比直接权重稀疏(SHiRA)具有更好的梯度覆盖性,且避免全局傅里叶基(FourierFT)的破坏性干扰。我们提供了理论分析:(i) 稀疏方法可实现高秩更新,突破LoRA的子空间瓶颈,提升表征能力;(ii) 提出梯度覆盖框架,解释何时更适合使用WaveFT。我们在文本到图像生成、图像分类和语言理解任务上进行了实验。结果表明,WaveFT在视觉任务中达到当前最优性能,因小波能有效捕捉稀疏梯度结构并提升覆盖性;在NLP任务上表现与现有方法相当。WaveFT已正式集成至Hugging Face PEFT库(huggingface.co/docs/peft/en/package_reference/waveft)。

原文摘要 · Abstract (English)

Efficiently adapting large pretrained models is critical under tight compute and memory budgets. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA achieve efficiency through low-rank updates, their discrete rank constraint limits fine-grained parameter control and confines adaptations to low-dimensional subspaces. We propose Wavelet Fine-Tuning (WaveFT), which learns sparse updates in the wavelet domain of weight matrices, enabling fine-grained control over trainable parameters well below LoRA's minimum rank. Wavelet bases provide semi-local receptive fields that aggregate spatially coherent gradients, offering better coverage than direct weight sparsity (SHiRA) without the destructive interference of global Fourier bases (FourierFT). We provide theoretical analysis showing: (i) sparse methods achieve high-rank updates, avoiding LoRA's subspace bottleneck and enabling higher representational capacity, and (ii) a gradient coverage framework explaining when WaveFT is preferable. We perform experiments across text-to-image generation, image classification, and language understanding. WaveFT demonstrates state-of-the-art results among PEFT methods for vision tasks, where wavelets effectively capture sparse gradient structure through improved coverage, while performing comparably on NLP tasks. WaveFT has officially been included in the Hugging Face PEFT library (huggingface.co/docs/peft/en/package_reference/waveft).

小波变换参数高效视觉微调稀疏更新

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。