一种新型概率回归模型,能同时处理点预测和不确定性估计。
Anchor-MoE: A Mean-Anchored Mixture of Experts For Probabilistic Regression
- 用锚定均值引导专家网络,通过软路由分配样本到局部专家
- 在多个数据集上优于NGBoost,RMSE和负对数似然双优
- 理论证明收敛率最优,适合需要可靠置信度的科研与工程场景
不确定性下的回归是科学与工程中的基础问题。本文提出锚定混合专家(Anchor-MoE)模型,同时支持概率回归与点回归。使用调优的梯度提升模型生成锚定均值,也可替换为任意现成的点回归器。锚定预测被投影至潜在空间,通过可学习的度量窗口核计算局部性,软路由将每个样本分发给一组混合密度网络专家;专家输出异方差修正与预测方差。训练目标为最小化负对数似然,并在独立校准集上拟合后验线性映射以提升点预测精度。理论上,在霍尔德光滑度为α、固定利普希茨权函数且重叠有界的条件下,锚定混合专家达到最优的L²风险率O(N^{-2α/(2α+d)})。CRPS测试泛化误差界为~O(√((log(Mh)+P+K)/N)),在有界重叠路由下可将K替换为k,隐维度影响被吸收进P。当均值与方差有界时,测试NLL同样具有类似√((log(Mh)+P+K)/N)的缩放。实验表明,在标准UCI回归任务中,锚定混合专家持续优于或超越强基线NGBoost,在多项数据集上达到新的概率回归最先进水平。代码已公开于https://github.com/BaozhuoSU/Probabilistic_Regression。
原文摘要 · Abstract (English)
Regression under uncertainty is fundamental across science and engineering. We present an Anchored Mixture of Experts (Anchor-MoE), a model that handles both probabilistic and point regression. For simplicity, we use a tuned gradient-boosting model to furnish the anchor mean; however, any off-the-shelf point regressor can serve as the anchor. The anchor prediction is projected into a latent space, where a learnable metric-window kernel scores locality and a soft router dispatches each sample to a small set of mixture-density-network experts; the experts produce a heteroscedastic correction and predictive variance. We train by minimizing negative log-likelihood, and on a disjoint calibration split fit a post-hoc linear map on predicted means to improve point accuracy. On the theory side, assuming a Hölder smooth regression function of order~$α$ and fixed Lipschitz partition-of-unity weights with bounded overlap, we show that Anchor-MoE attains the minimax-optimal $L^2$ risk rate $O\!\big(N^{-2α/(2α+d)}\big)$. In addition, the CRPS test generalization gap scales as $\widetilde{O}\!\Big(\sqrt{(\log(Mh)+P+K)/N}\Big)$; it is logarithmic in $Mh$ and scales as the square root in $P$ and $K$. Under bounded-overlap routing, $K$ can be replaced by $k$, and any dependence on a latent dimension is absorbed into $P$. Under uniformly bounded means and variances, an analogous $\widetilde{O}\!\big(\sqrt{(\log(Mh)+P+K)/N}\big)$ scaling holds for the test NLL up to constants. Empirically, across standard UCI regressions, Anchor-MoE consistently matches or surpasses the strong NGBoost baseline in RMSE and NLL; on several datasets it achieves new state-of-the-art probabilistic regression results on our benchmark suite. Code is available at https://github.com/BaozhuoSU/Probabilistic_Regression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。