用符号函数加速分布式学习收敛,但会牺牲精度,需权衡速度与误差。
Using Non-Lipschitz Signum-based Functions for Distributed Optimization and Machine Learning: Trade-off Between Con-vergence Rate and Optimality Gap
- 采用非Lipschitz符号函数提升分布式优化收敛速度
- 实验显示该方法收敛更快,但最终解与最优值差距大
- 适合对收敛速度敏感、可容忍一定误差的场景
近年来,大规模数据集和复杂学习模型的需求推动了高效分布式机器学习的发展。收敛速度是影响其实际应用效果的关键因素。近期提出的非Lipschitz连续优化算法旨在改善现有线性方法收敛慢的问题。符号函数在共识与控制领域被用于实现预定时间快速收敛,并对噪声和异常值具有鲁棒性。然而,本文发现这类算法在离散时间设置下会导致目标函数的最优性间隙和稳态残差。为此,我们研究了分布式优化与机器学习中收敛速度与最优性间隙之间的权衡。具体以分布式回归问题为例,对比了线性和非Lipschitz符号函数方法的收敛性能。通过大量仿真验证,结果表明:虽然符号函数能加快收敛,但会引入较大的最优性间隙。这些发现有助于推进分布式约束优化与分布式估计等类似算法的研究。
原文摘要 · Abstract (English)
In recent years, the prevalence of large-scale data-sets and the demand for sophisti-cated learning models have necessitated the development of efficient distributed ma-chine learning (ML) solutions. Convergence speed is a critical factor influencing the practicality and effectiveness of these distributed frameworks. Recently, non-Lipschitz continuous optimization algorithms have been proposed to improve the slow conver-gence rate of the existing linear solutions. The use of signum-based functions is previ-ously considered in consensus and control literature to reach fast convergence in the prescribed time and also to provide robust algorithms to noisy/outlier data. However, as shown in this work, these algorithms lead to an optimality gap and steady-state re-sidual of the objective function in discrete-time setup. This motivates us to investigate the distributed optimization and ML algorithms in terms of trade-off between conver-gence rate and optimality gap. In this direction, we specifically consider the distributed regression problem and check its convergence rate by applying both linear and non-Lipschitz signum-based functions. We check our distributed regression approach by extensive simulations. Our results show that although adopting signum-based func-tions may give faster convergence, it results in large optimality gaps. The findings pre-sented in this paper may contribute to and advance the ongoing discourse of similar distributed algorithms, e.g., for distributed constrained optimization and distributed estimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。