arXiv:2412.07242stat.MLcs.LG2024-12NeurIPS被引 2

用优化方法替代随机投影,实现更优的降维嵌入。

Optimization Can Learn Johnson Lindenstrauss Embeddings

  • 通过优化随机采样器的方差,避开传统投影矩阵的非凸陷阱。
  • 在保持距离不变的前提下,达到与随机投影相当甚至更好的压缩效果。
  • 适合对数据结构敏感、追求确定性结果的研究者使用。

嵌入在多个领域中扮演关键角色,能以紧凑形式表示复杂数据结构。随机方法如 Johnson-Lindenstrauss (JL) 提供了当前最优且理论上无法超越的保证,但其分析和算法未利用数据的潜在结构信息。自然的问题是:是否必须随机化?能否直接基于数据使用优化方法?初步答案是否定的:我们证明,JL 的保距目标在投影矩阵空间中具有非凸景观,存在大量劣质驻点。但这并非终点。本文提出一种受扩散模型启发的新方法:不直接在投影矩阵空间优化,而是在更大的随机解采样器空间中优化,逐步降低采样器方差。我们证明,通过该大空间的演化,目标可收敛至确定性(零方差)解,从而避开劣质驻点。该方法也可视为一种基于优化的去随机化策略,我们认为该思想可推广至诸多其他问题。

原文摘要 · Abstract (English)

Embeddings play a pivotal role across various disciplines, offering compact representations of complex data structures. Randomized methods like Johnson-Lindenstrauss (JL) provide state-of-the-art and essentially unimprovable theoretical guarantees for achieving such representations. These guarantees are worst-case and in particular, neither the analysis, nor the algorithm, takes into account any potential structural information of the data. The natural question is: must we randomize? Could we instead use an optimization-based approach, working directly with the data? A first answer is no: as we show, the distance-preserving objective of JL has a non-convex landscape over the space of projection matrices, with many bad stationary points. But this is not the final answer. We present a novel method motivated by diffusion models, that circumvents this fundamental challenge: rather than performing optimization directly over the space of projection matrices, we use optimization over the larger space of random solution samplers, gradually reducing the variance of the sampler. We show that by moving through this larger space, our objective converges to a deterministic (zero variance) solution, avoiding bad stationary points. This method can also be seen as an optimization-based derandomization approach and is an idea and method that we believe can be applied to many other problems.

降维优化嵌入去随机化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。