提出新算法让百万亿参数的高斯混合模型训练速度提升十倍。
Sublinear Variational Optimization of Gaussian Mixture Models with Millions to Billions of Parameters
- 用变分近似结合因子分析混合模型，降低计算复杂度。
- 训练10亿参数模型仅需不到9小时，比现有方法快10倍。
- 适合需要大规模生成建模与零样本去噪的科研与工程场景。
高斯混合模型（GMM）是机器学习中常用模型，但对大数据量 $N$ 和高维数据 $D$ 的训练在计算上变得不可行。针对具有任意协方差的 GMM，本文提出一种高效变分近似方法，并与因子分析混合模型（MFAs）结合。对于含 $C$ 个成分的 GMM，新算法将每轮迭代复杂度从 $\mathcal{O}(NCD^2)$ 降至与 $D$ 线性、$NC$ 次线性相关。数值实验表明，该算法实现整个优化过程的次线性缩放；在大规模基准测试中，相比现有最优方法提速达一个数量级；作为概念验证，成功在约一亿张图像上训练了超过 100 亿参数的 GMM，单台顶级 CPU 上训练时间不足九小时；此外，其在零样本图像去噪任务中表现出色，实现领先的去噪速度且性能保持领先。
原文摘要 · Abstract (English)
Gaussian Mixture Models (GMMs) range among the most frequently used models in machine learning. However, training large, general GMMs becomes computationally prohibitive for datasets that have many data points $N$ of high-dimensionality $D$. For GMMs with arbitrary covariances, we here derive a highly efficient variational approximation, which is then integrated with mixtures of factor analyzers (MFAs). For GMMs with $C$ components, our proposed algorithm substantially reduces runtime complexity from $\mathcal{O}(NCD^2)$ per iteration to a complexity scaling linearly with $D$ and sublinearly with $NC$. In numerical experiments, we first validate that the complexity reduction results in a sublinear scaling for the entire GMM optimization process. Second, we show on large-scale benchmarks that the sublinear algorithm results in speed-ups of an order-of-magnitude compared to the state-of-the-art. Third, as a proof of concept, we finally train GMMs with over 10 billion parameters on about 100 million images, observing training times of less than nine hours on a single state-of-the-art CPU. Finally, and forth, we demonstrate the effectiveness of large-scale GMMs on the task of zero-shot image denoising, where sublinear training results in state-of-the-art denoising times while competitive denoising performance is maintained.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。