LoRA更新主要集中在低频成分，可大幅压缩存储且保持性能。
SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates
- 通过DCT分析发现，仅33%的低频系数承载90%能量。
- 保留10%系数使存储减少10倍，仅损失1.95分性能（SST-2）。
- 高频成分或为噪声，适合追求轻量化的模型微调场景。
我们对LoRA权重更新的谱结构进行了系统性实证研究。通过对BERT-base和RoBERTa-base在四个GLUE基准（SST-2、MNLI、CoLA、QQP）上训练得到的适配矩阵进行二维离散余弦变换（DCT）分析，发现LoRA更新普遍由低频成分主导：平均而言，仅33%的DCT系数就包含了90%的总谱能量。仅保留10%频率系数即可将适配器存储减少10倍，而在SST-2上性能仅下降1.95个百分点。值得注意的是，在k=50%时的频率掩码优于完整LoRA的3个模型-任务组合，表明高频成分可能充当适应噪声。进一步发现，RoBERTa-base在所有任务中均比BERT-base具有更强的谱可压缩性，且任务复杂度决定谱敏感性：自然语言推断任务需要更多频率资源，而情感分类则更少。后续的SVD-DCT相关性分析（皮尔逊r=0.906，p<1e-9）将这一33%的常数与梯度下降的谱动态（Olsen et al., 2025）联系起来，为该现象提供了理论依据。这些发现启发了一种新的参数高效微调设计原则：适配过程中的谱稀疏性。
原文摘要 · Abstract (English)
We present a systematic empirical study of the spectral structure of LoRA weight updates. Through 2D Discrete Cosine Transform (DCT) analysis of trained adaptation matrices across BERT-base and RoBERTa-base on four GLUE benchmarks (SST-2, MNLI, CoLA, QQP), we establish that LoRA updates are universally dominated by low-frequency components: on average, just 33% of DCT coefficients capture 90% of total spectral energy. Retaining only 10% of frequency coefficients reduces adapter storage by 10x while sacrificing only 1.95 percentage points on SST-2. Notably, frequency masking at k=50% improves over full LoRA on 3 of 8 model-task pairs, suggesting high-frequency components act as adaptation noise. We further discover that RoBERTa-base is systematically more spectrally compressible than BERT-base across all tasks, and that task complexity governs spectral sensitivity: NLI tasks require more frequency budget than sentiment classification. A subsequent SVD-DCT correlation analysis (Pearson r=0.906, p<1e-9) connects the empirical 33% constant to the spectral dynamics of SGD (Olsen et al., 2025), suggesting a theoretical grounding for this finding. These findings motivate a new design principle for PEFT: spectral sparsity in adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。