提出新算法加速弹性网线性模型训练,性能提升三倍。
Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models
- 用泰勒展开替代非线性运算,简化坐标下降计算流程。
- 通过可调参数s实现计算并行化,提升效率且不损失收敛性。
- 适合大规模线性建模场景,尤其对需要快速求解路径的用户有用。
我们提出一种新型增强循环坐标下降(ECCD)框架,用于求解带有弹性网正则化的广义线性模型,显著降低训练时间。通过在当前迭代点进行泰勒展开,避免了梯度计算中的非线性操作。该近似使坐标下降中的向量递推关系可被展开,转化为更高效的批量计算。实验表明,递推可由可调整的整数参数 $s$ 展开,当 $s > 1$ 时性能提升但不影响收敛性,$s = 1$ 时即为原版坐标下降。与块坐标下降相比,ECCD避免了收敛延迟和数值不稳定性。我们在 C++ 中使用 Eigen 实现该方法,加速线性代数运算。与现有最先进求解器对比,在多种基准数据集上的正则化路径变体平均提速达 $3\times$。代码开源:https://github.com/Yixiao-Wang-Stats/ECCD。
原文摘要 · Abstract (English)
We present a novel enhanced cyclic coordinate descent (ECCD) framework for solving generalized linear models with elastic net constraints that reduces training time in comparison to existing state-of-the-art methods. We redesign the CD method by performing a Taylor expansion around the current iterate to avoid nonlinear operations arising in the gradient computation. By introducing this approximation, we are able to unroll the vector recurrences occurring in the CD method and reformulate the resulting computations into more efficient batched computations. We show empirically that the recurrence can be unrolled by a tunable integer parameter, $s$, such that $s > 1$ yields performance improvements without affecting convergence, whereas $s = 1$ yields the original CD method. A key advantage of ECCD is that it avoids the convergence delay and numerical instability exhibited by block coordinate descent. Finally, we implement our proposed method in C++ using Eigen to accelerate linear algebra computations. Comparison of our method against existing state-of-the-art solvers shows consistent performance improvements of $3\times$ in average for regularization path variant on diverse benchmark datasets. Our implementation is available at https://github.com/Yixiao-Wang-Stats/ECCD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。