研究正则化如何影响非参数均值-方差回归的不确定性量化效果。
On the Effect of Regularization on Nonparametric Mean-Variance Regression
- 提出统计场论框架解释模型在信号与噪声间的选择机制。
- 发现正则化强度导致预测从完美拟合到恒定输出的突变相变。
- 实验验证在UCI和ClimSim数据集上具有稳健的校准性能，适合需要可靠不确定性的场景。
不确定性量化对机器学习中的决策与风险评估至关重要。均值-方差回归模型通过为每个数据点预测均值和残差噪声，提供了一种简单的不确定性量化方法。然而，过参数化的均值-方差模型面临信号-噪声模糊性问题，难以判断预测目标应归因于信号（均值）还是噪声（方差）。在极端情况下，模型可能完全拟合训练目标且残差噪声为零，或给出恒定、无信息的预测并将目标全归为噪声。我们观察到这两种极端之间存在明显的相变现象，其由模型正则化驱动。通过不同正则化水平的实证研究，揭示了重复运行间的显著变异性。为此，我们构建了一个统计场论框架，能准确捕捉该相变行为，并与实验结果一致。该分析将正则化超参数搜索空间从二维降低至一维，大幅减少计算成本。在UCI数据集和大规模ClimSim数据集上的实验表明，模型具备良好的校准性能，能有效量化预测不确定性。
原文摘要 · Abstract (English)
Uncertainty quantification is vital for decision-making and risk assessment in machine learning. Mean-variance regression models, which predict both a mean and residual noise for each data point, provide a simple approach to uncertainty quantification. However, overparameterized mean-variance models struggle with signal-to-noise ambiguity, deciding whether prediction targets should be attributed to signal (mean) or noise (variance). At one extreme, models fit all training targets perfectly with zero residual noise, while at the other, they provide constant, uninformative predictions and explain the targets as noise. We observe a sharp phase transition between these extremes, driven by model regularization. Empirical studies with varying regularization levels illustrate this transition, revealing substantial variability across repeated runs. To explain this behavior, we develop a statistical field theory framework, which captures the observed phase transition in alignment with experimental results. This analysis reduces the regularization hyperparameter search space from two dimensions to one, significantly lowering computational costs. Experiments on UCI datasets and the large-scale ClimSim dataset demonstrate robust calibration performance, effectively quantifying predictive uncertainty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。