用黎曼优化训练低秩模型参数，提升语言模型效率。
Riemannian Gradient Descent for Low-Rank Architectures
- 在低秩矩阵上设计五种几何结构与块矩阵变体
- 在小模型多头注意力中测试，未超越AdamW基线
- 开源实现，适合低秩架构研究者参考
本文研究针对低秩分解矩阵参数的黎曼优化技术，应用于现代深度学习场景。探索算法设计空间中的十个方案：两种低秩矩阵几何、三种低秩部分等距几何，以及这五种结构的块矩阵变体（因子在块行和块列间共享）。将方法应用于小型语言模型的多头注意力参数。经过学习率调优后，所提方法未能显著优于AdamW基线。代码已公开。
原文摘要 · Abstract (English)
We explore Riemannian optimization techniques for rank-factored matrix parameters, targeting contemporary deep learning applications. We examine ten points in the algorithm design space: two geometries for rank-$r$ matrices, three geometries for rank-$r$ partial isometries, and block-matrix variants of these five, where factors are shared across block-rows and block-columns. We apply our methods to the multihead attention parameters in small language models. After tuning learning rates, our methods do not conclusively outperform an AdamW baseline. Our implementations are available online.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。