用神经微分方程建模Transformer，揭示权重动态并支持灵活微调
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
- 将注意力与前馈层权重表示为连续层索引的函数，实现可微分的动态建模
- 发现特征值幅值随层加深而增大，挑战了现有理论中的权共享假设
- 支持针对不同架构约束的自适应微调，提升模型可解释性与灵活性
基于Transformer架构的大语言模型近年来取得显著进展，引发对其内部机制的广泛关注。本文提出一种新颖方法，利用高度灵活的非自治神经微分方程（Neural ODE）建模Transformer结构。我们通过神经网络参数化注意力与前馈模块的所有权重，将其表示为连续层索引的函数。通过对模型动力学进行谱分析，发现特征值幅值随层数增加而上升，挑战了现有理论中普遍采用的权重共享假设。同时，借助李雅普诺夫指数分析令牌级敏感性，增强模型可解释性。所提出的神经ODE Transformer在多种配置和数据集上表现媲美或优于标准Transformer，且具备灵活的微调能力，可适应不同架构约束。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) based on transformer architectures have sparked significant interest in understanding their inner workings. In this paper, we introduce a novel approach to modeling transformer architectures using highly flexible non-autonomous neural ordinary differential equations (ODEs). Our proposed model parameterizes all weights of attention and feed-forward blocks through neural networks, expressing these weights as functions of a continuous layer index. Through spectral analysis of the model's dynamics, we uncover an increase in eigenvalue magnitude that challenges the weight-sharing assumption prevalent in existing theoretical studies. We also leverage the Lyapunov exponent to examine token-level sensitivity, enhancing model interpretability. Our neural ODE transformer demonstrates performance comparable to or better than vanilla transformers across various configurations and datasets, while offering flexible fine-tuning capabilities that can adapt to different architectural constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。