手推Transformer反向传播，理清每一层如何影响输出
Deep learning for pedestrians: backpropagation in Transformers
- 用轻量索引无依赖方法推导自注意力、层归一化等层的梯度
- 给出LoRA微调层的显式梯度表达式，支持参数高效优化
- 适合想深入理解模型内部运作机制的研究者
本文延续此前对CNN反向传播的向量化推导，将相同方法应用于基于Transformer的下一个词预测架构。针对嵌入层、多头自注意力和层归一化等新结构，采用无索引轻量方法推导梯度表达式，并进一步提供LoRA层的梯度公式以说明参数高效微调机制。尽管自动求导工具已很成熟，但手动推导反向传播能揭示前向传播中数值流动的细节，加深对各操作如何影响最终输出的理解。文末附带一个最小化GPT类网络的完整PyTorch实现及所有梯度更新的解析表达式。
原文摘要 · Abstract (English)
This document is a follow-up to our previous paper dedicated to a vectorized derivation of backpropagation in CNNs. Following the same principles and notations already put in place there, we now focus on transformer-based next-token-prediction architectures. To this end, we apply our lightweight index-free methodology to new types of layers such as embedding, multi-headed self-attention and layer normalization. In addition, we also provide gradient expressions for LoRA layers to illustrate parameter-efficient fine-tuning. Why bother doing manual backpropagation when there are so many tools that do this automatically? Any gap in understanding of how values propagate forward will become evident when attempting to differentiate the loss function. By working through the backward pass manually, we gain a deeper intuition for how each operation influences the final output. A complete PyTorch implementation of a minimalistic GPT-like network is also provided along with analytical expressions for of all of its gradient updates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。