用可证明最优的组合权重,融合多种算法提升性能且超快。
Soft Learning

- 构建异构模型库,通过交叉验证非负最小二乘法找最优组合权值。
- 在37个数据集上70%任务排名第一,训练速度比深度网络快72-435倍。
- 无需调参和GPU,兼具解释性与可扩展性,适合追求高效通用模型的用户。
现代机器学习迫使从业者在强大但昂贵的深度网络与快速但有限的经典算法间二选一。本文提出软学习(Soft Learning)框架,维护一个包含线性模型、树集成、核机器和神经网络的异构专家库,并通过交叉验证的非负最小二乘法发现可证明最优的组合权重。该方法保证性能不低于其专家组合中的最佳加权方案,在仅用CPU的情况下训练速度比深度网络快72至435倍(跨测试配置),提供基于学习权重的内在可解释性,揭示何种算法范式最适配数据,且具备未来可扩展性:新增专家数学上保证性能不降反升。在37个数据集(25个分类,12个回归)上对比九种方法(包括CatBoost和调优后的深度网络),软学习在70%的任务中排名第一,平均排名最优(Friedman检验,p = 1.12 × 10^-12),是唯一同时在分类和回归任务中均表现优异的方法,且无需GPU或超参数调优。这些结果表明,应从‘哪种算法最好’转向‘最优组合是什么’——软学习以形式化保证回答了这一问题,适用于任何数据模态。
原文摘要 · Abstract (English)
Modern machine learning forces practitioners to choose between powerful but expensive deep networks and fast but limited classical algorithms. Here we introduce Soft Learning, a framework that maintains a library of heterogeneous specialists -- spanning linear models, tree ensembles, kernel machines, and neural networks -- and discovers provably optimal combination weights through cross-validated non-negative least squares. Soft Learning is guaranteed to match or exceed the best weighted combination of its specialists, trains over two orders of magnitude faster than deep networks on CPU alone (72-435x faster across tested configurations), provides inherent interpretability through learned weights that reveal which algorithmic paradigm best fits the data, and is future-proof: adding specialists is mathematically guaranteed to maintain or improve performance. Across 37 datasets (25 classification, 12 regression) against nine methods including CatBoost and tuned deep networks, Soft Learning ranks first on 70% of tasks, achieves the best mean rank (Friedman test, p = 1.12 x 10^-12), and is the only method to simultaneously excel at both classification and regression -- all without GPU hardware or hyperparameter tuning. These results suggest a paradigm shift from "which algorithm is best?" to "what is the provably optimal combination?" -- a question Soft Learning answers with formal guarantees for any data modality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。