解析零初始化注意力机制，证明非线性提示更优且可高效训练。
On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
- 将零初始化注意力与专家混合模型关联，理论推导最优提示与门控估计。
- 非线性提示在基准测试中显著优于线性提示，数据稀缺下仍表现稳定。
- 适用于低资源微调场景，尤其适合希望提升性能的LLM轻量优化者。
LLaMA-Adapter 近期成为高效微调 LLaMA 模型的技术，利用零初始化注意力实现训练稳定与性能提升。然而，该方法的理论基础仍不明确。本文提供严谨的理论分析，揭示零初始化注意力与专家混合模型之间的联系。证明线性与非线性提示及门控函数均可被最优估计，其中非线性提示更具灵活性，利于未来应用。在公开大模型基准上实证验证，非线性提示表现优于线性提示。即使在有限训练数据条件下，两类提示均持续超越原始注意力，凸显零初始化注意力的鲁棒性与适应能力。
原文摘要 · Abstract (English)
The LLaMA-Adapter has recently emerged as an efficient fine-tuning technique for LLaMA models, leveraging zero-initialized attention to stabilize training and enhance performance. However, despite its empirical success, the theoretical foundations of zero-initialized attention remain largely unexplored. In this paper, we provide a rigorous theoretical analysis, establishing a connection between zero-initialized attention and mixture-of-expert models. We prove that both linear and non-linear prompts, along with gating functions, can be optimally estimated, with non-linear prompts offering greater flexibility for future applications. Empirically, we validate our findings on the open LLM benchmarks, demonstrating that non-linear prompts outperform linear ones. Notably, even with limited training data, both prompt types consistently surpass vanilla attention, highlighting the robustness and adaptability of zero-initialized attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。