让神经网络权重的谱特性可学习，提升正交结构的灵活性。
ManifoldFlow: SPD-Relaxed Stiefel Layers with Learnable Singular Spectrum
- 在Stiefel流形上构建可学习的正定谱，保持正交基的同时允许奇异值变化
- 在序列、表格和图像任务中，性能优于固定谱的Stiefel层，尤其在语言模型投影中提升显著
- 适用于需要正交基但又需方向性增益/衰减的场景，非通用全连接层替代
正交与Stiefel层可对神经网络权重实现精确的谱控制，但其强约束是所有奇异值必须为1。许多需要正交基的场景仍需方向性衰减或放大。本文提出ManifoldFlow，是对固定谱Stiefel层的最小松弛：在保持权重矩阵基位于Stiefel流形的同时，通过W = Q S^{1/2}（Q^T Q = I，S为正定）学习一个有界正谱。由于W^T W = S，S的特征值即为权重矩阵的平方奇异值，因此特征值裁剪即可直接控制奇异值。在成对的序列、表格和图像实验中，该方法在使用Stiefel先验的设置下均优于固定谱版本，语言模型递归投影中的提升最为显著。卷积分类头的边界案例表明：ManifoldFlow并非通用密集层替代品，而是针对正交基为有效先验的场景所设计的可学习谱Stiefel松弛。当基应为正交时，其谱无需冻结。代码已开源。
原文摘要 · Abstract (English)
Orthogonal and Stiefel layers give neural weights exact spectral control, but they also impose a strong modeling constraint: all represented singular values are fixed at one. Many settings that benefit from an orthonormal basis still need direction-dependent attenuation or amplification. We introduce ManifoldFlow, a minimal relaxation of a fixed-spectrum Stiefel layer that keeps the basis on the Stiefel manifold while learning a bounded positive spectrum through W = Q S^{1/2}, with Q^T Q = I and S positive definite. Since W^T W = S, the eigenvalues of S are exactly the squared singular values of the realized weight, making eigenvalue clipping a direct singular-value control mechanism. Across paired sequence, tabular, and image experiments, the learnable SPD spectrum improves the fixed-spectrum Stiefel counterpart in the reported settings where the Stiefel prior is useful, with the largest gains in recurrent language-model projections. Boundary cases in convolutional classifier heads clarify the intended scope: ManifoldFlow is not a universal dense-layer replacement, but a spectrum-learnable Stiefel relaxation for settings where an orthonormal basis is a useful prior. When the basis should be orthonormal, its spectrum need not be frozen. Code available at https://github.com/Hik289/manifold_flow
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。