提出HORST优化器，让稀疏Transformer训练更稳定且高效。
HORST: Composing Optimizer Geometries for Sparse Transformer Training
- 将优化器步骤视为非交换算子，通过双曲镜像映射引入L1稀疏性偏好
- 在高稀疏度下显著优于AdamW，在视觉与语言任务中均表现优异
- 适合需要高效稀疏训练的Transformer模型研究者使用
稀疏化Transformer仍面临根本挑战：标准优化器难以同时促进稀疏性和保持训练稳定性。有效自适应优化器具有隐式的$L_{\infty}$偏差以保障稳定，但稀疏性需要$L_1$偏差。为此，我们提出一种优化器步骤的组合方法，将其形式化为非交换算子，以系统分析并整合其优化几何。由此得到的HORST（Hyperbolic Operator for Robust Sparse Training）是一种模块化优化器，继承自适应方法的稳定性，同时通过双曲镜像映射引入$L_1$稀疏性偏差。实验表明，HORST在视觉与语言任务的Transformer稀疏训练中均具实用性。其在所有稀疏度水平上均显著优于AdamW基线，尤其在高稀疏度时提升显著。
原文摘要 · Abstract (English)
Sparsifying transformers remains a fundamental challenge, as standard optimizers fail to simultaneously encourage sparsity and maintain training stability. Effective adaptive optimizers exhibit an implicit $L_{\infty}$ bias favoring stability, yet, sparsity requires an $L_1$ bias. To integrate sparsity, we propose a composition of optimizer steps, which we cast as non-commutative operators to analyze and combine their optimization geometry in a principled way. This yields HORST (Hyperbolic Operator for Robust Sparse Training), a modular optimizer that inherits stability from adaptive methods while inducing $L_1$ sparsity bias through a hyperbolic mirror map. Our experiments demonstrate its utility for sparse training of transformers on both vision and language tasks. HORST consistently and significantly outperforms AdamW baselines across all sparsity levels, with large gains at higher sparsity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。