arXiv:2409.12849cs.LG2024-09被引 10

用少量模型实现超大规模集成效果,提升准确率与泛化能力。

A Margin-Maximizing Fine-Grained Ensemble Method

  • 构建可学习的置信度矩阵,精准捕捉每个分类器在不同类别上的优势。
  • 设计基于边距的损失函数,仅用十分之一模型数量超越传统随机森林。
  • 提出高效优化算法,动态调整权重并最大化分类边界,适合资源受限场景。

集成学习在机器学习中取得显著成功,但其对大量基学习器的依赖限制了在资源受限环境中的应用。本文提出一种新型‘边际最大化细粒度集成方法’,通过精心优化少量学习器,实现性能超越大规模集成。我们设计了一个可学习的置信度矩阵,量化每个分类器对各类别的置信度,精确捕捉个体学习器的类别特异性优势。同时,引入基于边距的损失函数,利用 logsumexp 技术构建平滑且部分凸的优化目标,提升优化效率,促进收敛,并支持自适应置信度分配。我们证明该损失函数满足 Lipschitz 连续性,据此开发出高效的梯度优化算法,同步最大化边距并动态调整学习器权重。大量实验表明,本方法仅使用传统随机森林十分之一的基学习器,便超越其性能,并优于其他先进集成方法。

原文摘要 · Abstract (English)

Ensemble learning has achieved remarkable success in machine learning, but its reliance on numerous base learners limits its application in resource-constrained environments. This paper introduces an innovative "Margin-Maximizing Fine-Grained Ensemble Method" that achieves performance surpassing large-scale ensembles by meticulously optimizing a small number of learners and enhancing generalization capability. We propose a novel learnable confidence matrix, quantifying each classifier's confidence for each category, precisely capturing category-specific advantages of individual learners. Furthermore, we design a margin-based loss function, constructing a smooth and partially convex objective using the logsumexp technique. This approach improves optimization, eases convergence, and enables adaptive confidence allocation. Finally, we prove that the loss function is Lipschitz continuous, based on which we develop an efficient gradient optimization algorithm that simultaneously maximizes margins and dynamically adjusts learner weights. Extensive experiments demonstrate that our method outperforms traditional random forests using only one-tenth of the base learners and other state-of-the-art ensemble methods.

集成学习边缘最大化小样本集成置信度建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。