通过投影梯度到零空间，加速优化且适配复杂模型
Teleportation With Null Space Gradient Projection for Optimization Acceleration
- 将梯度投影至输入零空间，保持损失不变性
- 在多个数据集上验证，显著提升优化速度
- 适用于CNN、Transformer等复杂架构，计算成本低
随着模型复杂度和数据规模的持续增长，优化技术愈发关键。其中，飞行术（teleportation）作为一种新兴方法，通过在损失不变水平集内导航，寻找具有优良几何特性的参数，从而加速基于梯度下降的收敛。现有飞行术算法主要在多层感知机（MLPs）上表现有效，但拓展至卷积神经网络（CNNs）和Transformer等先进架构仍具挑战，且常伴随高昂计算开销。为此，本文提出一种新算法：将飞行术目标函数的梯度投影至输入零空间，有效保持飞行术在损失不变水平集内的操作，同时降低计算成本。该方法可轻松从MLPs推广至CNN、Transformer及其他先进架构。我们在多种基准数据集和优化器上验证了其有效性，展示了广泛的适用性。
原文摘要 · Abstract (English)
Optimization techniques have become increasingly critical due to the ever-growing model complexity and data scale. In particular, teleportation has emerged as a promising approach, which accelerates convergence of gradient descent-based methods by navigating within the loss invariant level set to identify parameters with advantageous geometric properties. Existing teleportation algorithms have primarily demonstrated their effectiveness in optimizing Multi-Layer Perceptrons (MLPs), but their extension to more advanced architectures, such as Convolutional Neural Networks (CNNs) and Transformers, remains challenging. Moreover, they often impose significant computational demands, limiting their applicability to complex architectures. To this end, we introduce an algorithm that projects the gradient of the teleportation objective function onto the input null space, effectively preserving the teleportation within the loss invariant level set and reducing computational cost. Our approach is readily generalizable from MLPs to CNNs, transformers, and potentially other advanced architectures. We validate the effectiveness of our algorithm across various benchmark datasets and optimizers, demonstrating its broad applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。