提出高效采样高维谱范数球内矩阵的方法，用于大模型优化。
Uniform Sampling from High-dimensional Spectral Norm Balls
- 基于理论证明，采样矩阵奇异值趋近于1
- 在高维下奇异值收敛，验证方法有效性
- 适合大语言模型等大规模矩阵采样场景
受机器学习优化应用的启发，本文研究从单位谱范数球中均匀采样矩阵的挑战。证明了随着矩阵维度增加，采样矩阵的所有奇异值几乎必然收敛至1。该结果为提出的简单采样方法提供了理论依据，该方法适用于现代大语言模型中的大规模矩阵。实验结果展示了奇异值的收敛性，以及精确采样与所提近似采样方法的有效性。
原文摘要 · Abstract (English)
Motivated by an application in machine learning optimization, this paper focuses on the challenges of sampling a matrix uniformly from the unit spectral norm ball. It is proven that all singular values of sampled matrices converge to 1 almost surely as the matrix dimensions increase. This result provides the theoretical justification for a proposed simple sampling method applicable for large dimension sizes matching matrices found in modern large language models. Experimental results demonstrate both the convergence of the singular values, as well as the exact and proposed approximate sampling methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。