用离散哈特利变换精炼关键频谱,实现高效低参微调。
SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation
- 通过离散哈特利变换筛选各层重要频谱成分
- 相比基线方法参数量减少75%以上,性能不降反升
- 适合资源受限场景下的大模型高效微调
低秩适配(LoRA)在微调大模型时虽有效降低可训练参数量,但在扩展至更大模型或复杂任务时仍面临计算与内存挑战。本文提出基于离散哈特利变换的稀疏频谱适配(SSH),通过初始权重经离散哈特利变换后选择最具信息量的频谱成分,在轻量级逆变换投影回空间域进行更新。在单模态(如语言理解与生成)和多模态(如视频-文本理解)任务上的大量实验表明,SSH优于现有参数高效微调方法,在显著降低计算成本与内存需求的同时保持甚至提升性能。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) has been demonstrated effective in reducing the trainable parameter number when fine-tuning a large foundation model (LLM). However, it still encounters computational and memory challenges when scaling to larger models or addressing more complex task adaptation. In this work, we introduce Sparse Spectrum Adaptation via Discrete Hartley Transformation (SSH), a novel approach that significantly reduces the number of trainable parameters while enhancing model performance. It selects the most informative spectral components across all layers, under the guidance of the initial weights after a discrete Hartley transformation (DHT). The lightweight inverse DHT then projects the spectrum back into the spatial domain for updates. Extensive experiments across both single-modality tasks such as language understanding and generation and multi-modality tasks such as video-text understanding demonstrate that SSH outperforms existing parameter-efficient fine-tuning (PEFT) methods while achieving substantial reductions in computational cost and memory requirements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。