不采样也能高效估算大模型输出期望，更准且更快。
Estimating the expected output of wide random MLPs more efficiently than sampling
- 用累积量和埃尔米特展开构建各层激活分布近似，避免逐样本计算。
- 在足够宽的网络中，达到相同误差所需浮点运算量远少于蒙特卡洛采样。
- 特别擅长估计罕见事件概率，适合高风险场景建模与训练。
机器学习中估算期望损失最常用的方法是采样、计算每条样本的损失并取均值。但采样并非最优。本文针对初始化状态下的MLP，提出一种无需运行样本通过网络即可估算其在高斯输入下期望输出的方法。通过累积量和埃尔米特展开，构建各层激活分布的近似表示。理论与实证表明，对于足够宽的网络，该方法在达到目标均方误差时所需的浮点运算量显著低于蒙特卡洛采样。此外，该方法在估计罕见事件概率方面表现优异，并可应用于模型训练。这些结果为降低模型灾难性尾部风险的概率提供了新路径。
原文摘要 · Abstract (English)
By far the most common way to estimate an expected loss in machine learning is to draw samples, compute the loss on each one, and take the empirical average. However, sampling is not necessarily optimal. Given an MLP at initialization, we show how to estimate its expected output over Gaussian inputs without running samples through the network at all. Instead, we produce approximate representations of the distributions of activations at each layer, leveraging tools such as cumulants and Hermite expansions. We show both theoretically and empirically that for sufficiently wide networks, our estimator achieves a target mean squared error using substantially fewer FLOPs than Monte Carlo sampling. We find moreover that our methods perform particularly well at estimating the probabilities of rare events, and additionally demonstrate how they can be used for model training. Together, these findings suggest a path to producing models with a greatly reduced probability of catastrophic tail risks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。