用PyTorch自动微分计算大模型困惑度的海森矩阵部分。
Hessian of Perplexity for Large Language Models by PyTorch autograd (Open Source)
- 基于PyTorch autograd计算LLM困惑度的海森矩阵局部信息。
- 通过多组向量-海森乘积估算完整对角线元素。
- 适合研究模型优化与参数敏感性的研究人员使用。
由于大型语言模型（LLM）规模庞大，计算其完整的海森矩阵（二阶导数矩阵）不可行。本文提供一份技术指南，展示如何利用PyTorch autograd库准确计算LLM困惑度的海森矩阵的一小部分。我们还演示了如何通过多个样本的向量-海森乘积（HVPs）来估计海森矩阵的完整对角线。我们希望本指南及配套的GitHub代码能为关注模型行为与结构的研究者和实践者提供有价值的参考。
原文摘要 · Abstract (English)
Computing the full Hessian matrix -- the matrix of second-order derivatives for an entire Large Language Model (LLM) is infeasible due to its sheer size. In this technical report, we aim to provide a comprehensive guide on how to accurately compute at least a small portion of the Hessian for LLMs using PyTorch autograd library. We also demonstrate how to compute the full diagonal of the Hessian matrix using multiple samples of vector-Hessian Products (HVPs). We hope that both this guide and the accompanying GitHub code will be valuable resources for practitioners and researchers interested in better understanding the behavior and structure of the Hessian in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。