从似然角度统一分析模型集成聚合,给出最优选择范围。
Beyond Mixtures and Products for Ensemble Aggregation: A Likelihood Perspective on Generalized Means
- 用广义均值视角统一分析集成聚合方法。
- 证明r∈[0,1]时能系统提升性能,r=0和r=1最可靠。
- 适用于需要稳定集成效果的场景,如图像/文本分类。
密度聚合是机器学习中的核心问题,例如在深度集成(Deep Ensemble)中融合多个预测结果。目前主流方法有线性池化(概率平均)和几何池化(对数几率平均),但聚合方式仍无定论。本文通过似然函数这一标准评估准则,研究了任意实数阶r∈ℝ∪{−∞,+∞}的归一化广义均值。该框架揭示了不同场景下的最优配置:当r∈[0,1]时,聚合结果可系统优于单个分布,为线性(r=1)与几何(r=0)池化提供了理论依据。相反,若r∉[0,1],则可能出现性能退化,本文以显式反例说明。最后,我们在图像与文本分类基准上使用深度集成进行了实证验证,支持了理论结论。
原文摘要 · Abstract (English)
Density aggregation is a central problem in machine learning, for instance when combining predictions from a Deep Ensemble. The choice of aggregation remains an open question with two commonly proposed approaches being linear pooling (probability averaging) and geometric pooling (logit averaging). In this work, we address this question by studying the normalized generalized mean of order $r \in \mathbb{R} \cup \{-\infty,+\infty\}$ through the lens of log-likelihood, the standard evaluation criterion in machine learning. This provides a unifying aggregation formalism and shows different optimal configurations for different situations. We show that the regime $r \in [0,1]$ is the only range ensuring systematic improvements relative to individual distributions, thereby providing a principled justification for the reliability and widespread practical use of linear ($r=1$) and geometric ($r=0$) pooling. In contrast, we show that aggregation rules with $r \notin [0,1]$ may fail to provide consistent gains with explicit counterexamples. Finally, we corroborate our theoretical findings with empirical evaluations using Deep Ensembles on image and text classification benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。