提出稀疏SVM的局部对偶理论,解释其优于传统模型的原因。
Local Duality for Sparse Support Vector Machines
- 基于局部对偶理论推导稀疏SVM,建立与0/1损失SVM的严格对应关系。
- 证明了线性表示定理在局部解上成立,且hSVM全局解可收敛到局部解。
- 为超参数选择提供依据,适合关注模型可解释性与性能的从业者。
由于基数最小化在优化中的兴起,稀疏支持向量机(SSVM)近年来备受关注,并展现出相对于凸SVM的某些经验优势。通常通过在凸SVM的对偶问题中加入基数函数(如ℓ₀-范数)来构建SSVM,但这一过程缺乏理论支撑。本文填补了这一空白,发展了此类SSVM公式的局部对偶理论,并探讨其与合页损失SVM(hSVM)和阶梯损失SVM(rSVM)的关系。特别地,我们证明所推导的SSVM正是0/1损失SVM的对偶问题,且其局部解满足线性表示定理。此外,在特定条件下,hSVM的一系列全局解会收敛至0/1损失SVM的局部解;同时,0/1损失SVM的一个局部极小点也是rSVM的局部极小点。这解释了为何先前研究中由SSVM诱导的局部解表现优于hSVM和rSVM。我们在真实数据集上进行了数值测试,验证了该论文提出的局部良好解在实际应用中的潜力。
原文摘要 · Abstract (English)
Due to the rise of cardinality minimization in optimization, sparse support vector machines (SSVMs) have attracted much attention lately and show certain empirical advantages over convex SVMs. A common way to derive an SSVM is to add a cardinality function such as $\ell_0$-norm to the dual problem of a convex SVM. However, this process lacks theoretical justification. This paper fills the gap by developing a local duality theory for such an SSVM formulation and exploring its relationship with the hinge-loss SVM (hSVM) and the ramp-loss SVM (rSVM). In particular, we prove that the derived SSVM is exactly the dual problem of the 0/1-loss SVM, and the linear representer theorem holds for their local solutions. The local solution of SSVM also provides guidelines on selecting hyperparameters of hSVM and rSVM. {Under specific conditions, we show that a sequence of global solutions of hSVM converges to a local solution of 0/1-loss SVM. Moreover, a local minimizer of 0/1-loss SVM is a local minimizer of rSVM.} This explains why a local solution induced by SSVM outperforms hSVM and rSVM in the prior empirical study. We further conduct numerical tests on real datasets and demonstrate potential advantages of SSVM by working with locally nice solutions proposed in this paper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。