用动态分解法提升模型优化器的矩阵近似精度
DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
- 引入投影分裂积分器构建可高效更新的矩阵近似
- 在大语言模型训练中显著优于现有优化器
- 适合需要高精度梯度预处理的深度学习场景
近期，将权重视为矩阵而非扁平向量的优化器展现出优异性能。这一视角自然催生了以结构化方式近似费舍尔信息矩阵作为预条件器的方法，其中矩阵视角带来克罗内克分解形式，实现内存高效表示。然而，高效且准确地构建此类近似仍是开放挑战，因为最优分解计算资源消耗大，实际方法多依赖启发式设计。本文提出新方法，利用投影分裂积分器构建有效预条件器。所提优化器 DyKAF（动态克罗内克近似费舍尔矩阵）持续提升费舍尔矩阵近似质量。在大规模语言模型预训练与微调实验中，DyKAF 在多种评估指标上均优于现有优化器。
原文摘要 · Abstract (English)
Recently, optimizers that explicitly treat weights as matrices, rather than flattened vectors, have demonstrated their effectiveness. This perspective naturally leads to structured approximations of the Fisher matrix as preconditioners, where the matrix view induces a Kronecker-factorized form that enables memory-efficient representation. However, constructing such approximations both efficiently and accurately remains an open challenge, since obtaining the optimal factorization is resource-intensive and practical methods therefore rely on heuristic design choices. In this work, we introduce a novel approach that leverages projector-splitting integrators to construct effective preconditioners. Our optimizer, DyKAF (Dynamical Kronecker Approximation of the Fisher Matrix), consistently improves the Fisher matrix approximation quality. Experiments on large language model pre-training and fine-tuning demonstrate that DyKAF outperforms existing optimizers across a range of evaluation metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。