证明对比学习找局部最优解是计算上极难的,意味着算法可能永远卡住。
The Complexity of Finding Local Optima in Contrastive Learning
- 在离散和连续场景下分别证明了局部最优解的计算复杂性
- 无论是否在低维空间,局部搜索都可能需要指数时间
- 对现有对比学习算法的收敛性提供了理论限制
对比学习通过优化基于对比信息的目标函数来发现有意义的数据表示,通常以加权三元组集合 $\\(\{(x_i, y_i^+, z_{i}^-)\ }_{i = 1}^m$ 形式给出,表示锚点 $x_i$ 应比负例 $z_i$ 更接近正例 $y_i$。目标是寻找使锚点与正例更接近、与负例更远的表示(如 $\\\mathbb{R}^d$ 中的嵌入或树度量)。虽然全局最优解的求解已被证明为 $\\\mathsf{NP}$-难,但局部最优解的复杂性仍未知。本文通过证明离散设置下(如最大化满足三元组数)为 $\\\mathsf{PLS}$-难,连续设置下(如最小化三元组损失)为 $\\\mathsf{CLS}$-难,解决了该问题。结果表明,除非 $\\\mathsf{PLS}\\subseteq \\\mathsf{P}$(或 $\\\mathsf{CLS}\\subseteq \\\mathsf{P}$),否则不存在多项式时间算法能求得局部最优解。即使假设成立,我们的归约也表明存在实例,使得局部搜索算法在 $d=1$ 时仍需指数时间才能收敛。
原文摘要 · Abstract (English)
Contrastive learning is a powerful technique for discovering meaningful data representations by optimizing objectives based on $\textit{contrastive information}$, often given as a set of weighted triplets $\{(x_i, y_i^+, z_{i}^-)\}_{i = 1}^m$ indicating that an "anchor" $x_i$ is more similar to a "positive" example $y_i$ than to a "negative" example $z_i$. The goal is to find representations (e.g., embeddings in $\mathbb{R}^d$ or a tree metric) where anchors are placed closer to positive than to negative examples. While finding $\textit{global}$ optima of contrastive objectives is $\mathsf{NP}$-hard, the complexity of finding $\textit{local}$ optima -- representations that do not improve by local search algorithms such as gradient-based methods -- remains open. Our work settles the complexity of finding local optima in various contrastive learning problems by proving $\mathsf{PLS}$-hardness in discrete settings (e.g., maximize satisfied triplets) and $\mathsf{CLS}$-hardness in continuous settings (e.g., minimize Triplet Loss), where $\mathsf{PLS}$ (Polynomial Local Search) and $\mathsf{CLS}$ (Continuous Local Search) are well-studied complexity classes capturing local search dynamics in discrete and continuous optimization, respectively. Our results imply that no polynomial time algorithm (local search or otherwise) can find a local optimum for various contrastive learning problems, unless $\mathsf{PLS}\subseteq\mathsf{P}$ (or $\mathsf{CLS}\subseteq \mathsf{P}$ for continuous problems). Even in the unlikely scenario that $\mathsf{PLS}\subseteq\mathsf{P}$ (or $\mathsf{CLS}\subseteq \mathsf{P}$), our reductions imply that there exist instances where local search algorithms need exponential time to reach a local optimum, even for $d=1$ (embeddings on a line).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。