提出稀疏化核逻辑回归新方法,兼顾预测精度与模型简洁性。
Binary Kernel Logistic Regression: a sparsity-inducing formulation and a convergent decomposition training algorithm
- 基于改进的核逻辑回归公式,主动诱导模型稀疏
- 在12个数据集上实现高精度与强稀疏性的平衡
- 算法收敛且适合需要概率输出的场景
核逻辑回归(KLR)是一种广泛用于二分类和多分类的监督学习方法,可估计数据点的类别归属条件概率。与支持向量机(SVM)等核方法不同,传统KLR通常不稀疏。此前尝试通过引入向量机(IVM)等启发式方法或ℓ₁/₂正则化来提升稀疏性,但准确率与稀疏性的权衡仍具挑战。本文重新审视二分类KLR,提出一种扩展训练公式,可在保持良好测试精度的同时诱导模型稀疏。为高效求解该公式的对偶问题,设计了一种类似序列最小优化(SMO)的分解算法,利用二阶信息并证明其全局收敛性。在12个公开数据集上的实验表明,所提方法在精度与稀疏性之间达到良好平衡,优于IVM、ℓ₁/₂正则化KLR及SVM,同时保留了提供类概率估计的优势。
原文摘要 · Abstract (English)
Kernel logistic regression (KLR) is a widely used supervised learning method for binary and multi-class classification, which provides estimates of the conditional probabilities of class membership for the data points. Unlike other kernel methods such as Support Vector Machines (SVMs), KLRs are generally not sparse. Previous attempts to deal with sparsity in KLR include a heuristic method referred to as the Import Vector Machine (IVM) and ad hoc regularizations such as the $\ell_{1/2}$-based one. Achieving a good trade-off between prediction accuracy and sparsity is still a challenging issue with a potential significant impact from the application point of view. In this work, we revisit binary KLR and propose an extension of the training formulation proposed by Keerthi et al., which is able to induce sparsity in the trained model, while maintaining good testing accuracy. To efficiently solve the dual of this formulation, we devise a decomposition algorithm of Sequential Minimal Optimization type which exploits second-order information, and for which we establish global convergence. Numerical experiments conducted on 12 datasets from the literature show that the proposed binary KLR approach achieves a competitive trade-off between accuracy and sparsity with respect to IVM, $\ell_{1/2}$-based regularization for KLR, and SVM while retaining the advantages of providing informative estimates of the class membership probabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。