通过谱对齐提升低秩适配器的压缩效率,让模型在更少参数下保持高性能。
SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation

- 设计可微分角度损失,主动对齐学生模型权重子空间。
- 在合成数据上将子空间错位降至近零,最终对齐度达1.00。
- 适用于追求高参数效率的NLP模型压缩任务。
将微调后的教师模型蒸馏到LoRA适配的学生模型中是参数高效压缩的标准方法,但输出级知识蒸馏无法显式控制适配器占据的秩-$r$权重子空间。本文提出SAD-LoRA(谱对齐蒸馏),从数据加权的学生空间参考更新$oldsymbol{D}_W oldsymbol{ ilde{ heta}}^{1/2}$中选取该子空间,并通过可微分主角损失在训练中维持其稳定性。我们证明数据加权蒸馏误差可精确分解为子空间错位、子空间内系数不匹配和不可约秩残差;标准蒸馏仅能间接影响第一项。在教师谱平坦的可控合成问题中,SAD-LoRA将子空间错位从51%降至近乎零,最终子空间对齐度由0.49提升至1.00。在六项GLUE任务上,从RoBERTa-large到RoBERTa-base的蒸馏中,SAD-LoRA在$ r=4 $时于五项任务上达到或超越最强谱基线,在$ r=8 $时在SST-2和CoLA上取得最优结果。消融实验表明子空间对齐是核心贡献,系数匹配仅为辅助。
原文摘要 · Abstract (English)
Distilling a fine-tuned teacher into a LoRA-adapted student is a standard recipe for parameter-efficient compression, but output-level KD does not explicitly control which rank-$r$ weight subspace the adapter occupies. We propose \textbf{SAD-LoRA} (\textbf{S}pectral \textbf{A}lignment \textbf{D}istillation), which selects this subspace from the data-weighted student-space reference update $\DWT\Sigx^{1/2}$ and maintains it during training via a differentiable principal-angle loss on $\colspan(B)$. We show that the data-weighted distillation error decomposes exactly into subspace misalignment, within-subspace coefficient mismatch, and irreducible rank residual; standard KD can affect the first term only indirectly through output gradients. On controlled synthetic problems with a flat teacher spectrum, SAD-LoRA reduces the subspace-misalignment term from $51\%$ to nearly zero and lifts final subspace alignment from $0.49$ to $1.00$. On RoBERTa-large to RoBERTa-base distillation across six GLUE tasks, SAD-LoRA improves rank efficiency: at $r{=}4$, it matches or beats the strongest included spectral baseline on five of six tasks, and at $r{=}8$ it gives the best result on SST-2 and CoLA. Ablations identify subspace alignment as the load-bearing component, while coefficient matching is auxiliary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。