证明softmax在所有ℓp范数下最大光滑度为1/2,比传统认知更紧。
Softmax is $1/2$-Lipschitz: A tight bound across all $\ell_p$ norms
- 首次统一分析softmax在所有ℓp范数下的光滑性
- 理论证明其全局Lipschitz常数严格不超过1/2
- 适用于关注模型鲁棒性与收敛性的研究者
Softmax函数是机器学习与优化中的基础算子,广泛应用于分类、注意力机制、强化学习和涉及log-sum-exp的场景。现有理论通常假设softmax关于ℓ₂范数的Lipschitz常数为1。本文证明,softmax在所有ℓ_p范数(p ≥ 1)下均为收缩映射,且统一的Lipschitz常数为1/2。我们进一步表明,当p = 1或p = ∞时,局部Lipschitz常数可达1/2;而对于p ∈ (1,∞),该常数严格小于1/2,且仅在极限情况下趋近于1/2。据我们所知,这是首个对softmax Lipschitz连续性进行范数统一分析的工作。我们展示了更紧的1/2常数如何直接改进现有鲁棒性与收敛性理论结果,并通过在ViT、GPT-2和Qwen3-8B等注意力模型以及强化学习中的随机策略上的实证研究验证了该常数的紧致性。
原文摘要 · Abstract (English)
The softmax function is a basic operator in machine learning and optimization, used in classification, attention mechanisms, reinforcement learning, game theory, and problems involving log-sum-exp terms. Existing robustness guarantees of learning models and convergence analysis of optimization algorithms typically consider the softmax operator to have a Lipschitz constant of $1$ with respect to the $\ell_2$ norm. In this work, we prove that the softmax function is contractive with the Lipschitz constant $1/2$, uniformly across all $\ell_p$ norms with $p \ge 1$. We also show that the local Lipschitz constant of softmax attains $1/2$ for $p = 1$ and $p = \infty$, and for $p \in (1,\infty)$, the constant remains strictly below $1/2$ and the supremum $1/2$ is achieved only in the limit. To our knowledge, this is the first comprehensive norm-uniform analysis of softmax Lipschitz continuity. We demonstrate how the sharper constant directly improves a range of existing theoretical results on robustness and convergence. We further validate the sharpness of the $1/2$ Lipschitz constant of the softmax operator through empirical studies on attention-based architectures (ViT, GPT-2, Qwen3-8B) and on stochastic policies in reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。