用奇异值分解优化大模型微调，参数少25%且收敛更快。
Singular Value Decomposition on Kronecker Adaptation for Large Language Model
- 通过奇异值分解动态生成克罗内克因子，自动选择最优秩。
- 仅需0.99M可训练参数，比LoRA/PiSSA少25%，性能相当或更优。
- 适合资源受限场景下的大模型高效微调，尤其擅长数学与代码任务。
大规模预训练Transformer模型在多种语言和推理任务中表现卓越，但全量微调带来巨大的存储、内存和计算开销。参数高效微调（PEFT）方法通过仅学习少量任务相关参数来缓解此问题，但现有方法或引入推理延迟（适配器模块），或收敛效果不佳（随机初始化低秩更新），或依赖固定秩选择而无法匹配任务复杂度（基于克罗内克分解）。本文提出SoKA（SVD on Kronecker Adaptation），一种结合克罗内克乘积张量分解与SVD驱动初始化及谱感知动态秩选择的新型PEFT策略。其克罗内克乘积奇异值分解（KPSVD）过程将完整权重更新的主成分提取为紧凑的克罗内克因子，自适应秩选择算法则利用能量阈值与肘点判据剔除冗余分量。在LLaMA2-7B上对算术推理（GSM8K）、形式数学（MATH）和代码生成（MBPP）的任务评估表明，SoKA仅需0.99M可训练参数，比LoRA/PiSSA减少25%，同时达到或超越基线性能。此外，SoKA展现出更快的收敛速度和更稳定的梯度，凸显其在大规模模型适配中的鲁棒性与高效性。
原文摘要 · Abstract (English)
Large pre-trained Transformer models achieve state-of-the-art results across diverse language and reasoning tasks, but full fine-tuning incurs substantial storage, memory, and computational overhead. Parameter-efficient fine-tuning (PEFT) methods mitigate these costs by learning only a small subset of task-specific parameters, yet existing approaches either introduce inference-time latency (adapter modules), suffer from suboptimal convergence (randomly initialized low-rank updates), or rely on fixed rank choices that may not match task complexity (Kronecker-based decompositions). We propose SoKA (SVD on Kronecker Adaptation), a novel PEFT strategy that combines Kronecker-product tensor factorization with SVD-driven initialization and spectrum-aware dynamic rank selection. Our Kronecker-Product SVD (KPSVD) procedure extracts principal components of the full weight update into compact Kronecker factors, while an adaptive rank selection algorithm uses energy-threshold and elbow-point criteria to prune negligible components. Empirical evaluation on LLaMA2-7B across arithmetic reasoning (GSM8K), formal mathematics (MATH), and code generation (MBPP) demonstrates that SoKA requires only 0.99M trainable parameters, 25% fewer than LoRA/PiSSA, while matching or exceeding baseline performance. Moreover, SoKA exhibits faster convergence and more stable gradients, highlighting its robustness and efficiency for large-scale model adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。