通过谱分析发现大模型领域适配主要靠向量变化，可删60%低重要性头提升效率。
Diffract: Spectral View of LLM Domain Adaptation
- 用奇异值分解分析权重矩阵，发现领域适配依赖奇异向量而非谱分布变化。
- 识别出注意力头的领域特异性差异，可移除最多60%头更新而不影响性能。
- 选择性重置低重要性头使基准测试准确率提升最高4%，适合高效微调场景。
我们研究持续预训练（CPT）作为将通用大语言模型适配至数学、指令、代码和自然文本等专用领域的机制。通过对权重矩阵进行奇异值分解，发现CPT使奇异值谱保持基本不变，适应主要由奇异向量变化驱动。对注意力头投影矩阵的分析揭示了显著的领域依赖性头异质性，据此提出头重要性准则：最多可移除60%的头更新而无明显质量损失。选择性将低重要性头回滚至预训练状态，相较于完全训练基线，可使基准测试准确率提升最高4%。最后，我们识别出领域连通性——在CPT检查点间进行线性插值可实现平滑的领域-质量过渡，且两领域均无明显退化。我们发布Diffract，一个开源工具包，用于对十亿参数模型进行可扩展的谱分析。
原文摘要 · Abstract (English)
We study continual pre-training (CPT) as a mechanism for adapting general-purpose large language models to specialized domains: mathematics, instruction, code, and natural text. Using singular value decomposition of weight matrices, we find that CPT leaves singular value spectra largely invariant, with adaptation driven mainly by changes in singular vectors. An analysis of attention-head projection matrices reveals strong, domain-dependent head heterogeneity, which we exploit to define a head importance criterion: up to 60% of head updates can be removed without measurable quality loss. Selectively rewinding low-importance heads to their pre-trained state improves benchmark accuracy by up to 4% versus the fully trained baseline. Finally, we identify domain connectivity - linear interpolation between CPT checkpoints yields smooth domain-quality interpolation without notable degradation on either domain - and release Diffract, an open-source toolkit for scalable spectral analysis of billion-parameter models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。