提升高秩DoRA效率，解决显存瓶颈并加速计算。
Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels
- 通过分解范数计算避免密集矩阵乘法，降低内存占用。
- 融合核函数使推理与梯度计算提速1.5至2倍，显存节省达7GB。
- 适合在单卡上部署高秩视觉语言模型微调的研究者使用。
权重分解低秩适配（DoRA）通过分离权重大小与方向扩展了LoRA，但其前向传播需计算W + sBA的行范数，现有框架均通过显式生成稠密的[d_out, d_in]矩阵BA实现。当d_in = 8192、rank r = 384时，单个模块的范数计算需约512 MB临时显存（bf16精度），在数百个适配模块和检查点启用下，高秩DoRA在主流单卡设置中变得昂贵且不可行。本文提出两项系统性改进：因子化范数将平方范数分解为基项、交叉项和格拉姆项，仅需O(d_out r + r^2)中间结果即可计算，消除稠密乘积；融合Triton内核将原本四次核函数调用合并为一次，减少约4倍内存传输，并采用数值稳定形式，在接近单位缩放的实用场景中避免灾难性抵消。在三款NVIDIA GPU（RTX 6000 PRO, H200, B200）上对六组8-32B视觉语言模型（VLMs）进行测试，当r = 384、bf16精度下，融合实现相比Hugging Face PEFT的DoRA在推理上提速1.5-2.0倍，梯度计算提速1.5-1.9倍（不含优化器步骤），峰值显存降低最高达7 GB。跨六款GPU（涵盖四代架构：L40S, A100, RTX 6000 PRO, H200, B200, B300）的微基准测试验证了1.5-2.7倍的组合核加速。所有模型/硬件组合的最终输出余弦相似度超过0.9999，多种子训练曲线在2000步内平均每步损失差异小于7.1×10⁻⁴。
原文摘要 · Abstract (English)
Weight-Decomposed Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction, but its forward pass requires the row-wise norm of W + sBA, a computation that every major framework we surveyed implements by materializing the dense [d_out, d_in] product BA. At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16, making high-rank DoRA costly and often infeasible on common single-GPU setups once hundreds of adapted modules and checkpointing are involved. We present two systems contributions. A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r^2) intermediates, eliminating the dense product. Fused Triton kernels collapse the four-kernel DoRA composition into a single pass, reducing memory traffic by about 4x and using a numerically stable form that avoids catastrophic cancellation in the near-unity rescaling regime where magnitude scales concentrate in practice. Across six 8-32B vision-language models (VLMs) on three NVIDIA GPUs (RTX 6000 PRO, H200, B200) at r = 384 in bf16, the fused implementation is 1.5-2.0x faster than Hugging Face PEFT's DoRA implementation for inference and 1.5-1.9x faster for gradient computation (optimizer step excluded), with up to 7 GB lower peak VRAM. Microbenchmarks on six GPUs spanning four architecture generations (L40S, A100, RTX 6000 PRO, H200, B200, B300) confirm 1.5-2.7x compose-kernel speedup. Final-logit cosine similarity exceeds 0.9999 across all model/GPU pairs, and multi-seed training curves match within 7.1 x 10^-4 mean per-step loss delta over 2000 steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。