突破百亿参数模型的海森矩阵计算瓶颈，实现单机多卡分布式计算。
HessFormer: Hessians at Foundation Scale
- 基于分布式随机兰姆齐算法，支持单机多卡并行计算海森向量积。
- 首次在700亿参数的DeepSeek模型上完成海森谱密度分析。
- 开源工具包兼容Transformers，适合研究大模型优化与二阶信息的学者。
尽管深度学习模型的一阶优化已取得显著进展，当前开源混合专家模型参数量已达数百亿级别，但依赖海森向量积的方法仍受限于单个GPU，无法处理十亿参数以上的模型。本文发布软件包HessFormer，可无缝集成至主流Transformers库，实现单机多卡环境下的分布式海森向量计算。其核心是公开发布的分布式随机兰姆齐求积算法。利用该工具，我们对最新的700亿参数Deepseek模型进行了海森谱密度分析。
原文摘要 · Abstract (English)
Whilst there have been major advancements in the field of first order optimisation of deep learning models, where state of the art open source mixture of expert models go into the hundreds of billions of parameters, methods that rely on Hessian vector products, are still limited to run on a single GPU and thus cannot even work for models in the billion parameter range. We release a software package \textbf{HessFormer}, which integrates nicely with the well known Transformers package and allows for distributed hessian vector computation across a single node with multiple GPUs. Underpinning our implementation is a distributed stochastic lanczos quadrature algorithm, which we release for public consumption. Using this package we investigate the Hessian spectral density of the recent Deepseek $70$bn parameter model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。