提出更优的随机迹估计法，提升生成模型训练稳定性与质量
Optimal Stochastic Trace Estimation in Generative Modeling
- 引入Hutch++算法，降低生成模型训练中的方差
- 在高维低秩数据下显著减少方差，生成图像质量更高
- 适合需要稳定训练的扩散模型、时间序列预测等场景
Hutchinson估计器广泛用于扩散模型的基于散度的似然训练，以保证最优传输（OT）特性。然而该估计器常因高方差和可扩展性问题而受限。为此，本文研究了适用于生成模型的最优随机迹估计器Hutch++，旨在最小化训练方差的同时保持运输最优性。该方法对条件数大的病态矩阵特别有效，这在高维数据具有低维结构时常见。为避免频繁且昂贵的QR分解，我们提出了兼顾频率与精度的实际方案，并提供理论保证。分析表明，Hutch++能生成更高质量的样本。此外，该方法在模拟、条件时间序列预测和图像生成等多种应用中均表现出显著的方差降低效果。
原文摘要 · Abstract (English)
Hutchinson estimators are widely employed in training divergence-based likelihoods for diffusion models to ensure optimal transport (OT) properties. However, this estimator often suffers from high variance and scalability concerns. To address these challenges, we investigate Hutch++, an optimal stochastic trace estimator for generative models, designed to minimize training variance while maintaining transport optimality. Hutch++ is particularly effective for handling ill-conditioned matrices with large condition numbers, which commonly arise when high-dimensional data exhibits a low-dimensional structure. To mitigate the need for frequent and costly QR decompositions, we propose practical schemes that balance frequency and accuracy, backed by theoretical guarantees. Our analysis demonstrates that Hutch++ leads to generations of higher quality. Furthermore, this method exhibits effective variance reduction in various applications, including simulations, conditional time series forecasts, and image generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。