提出一种可间歇跳过计算的去中心化双层优化算法,提升AI系统协作效率。
S$^3$LDBO: A Snapshot Single-Loop Algorithm for Decentralized Bilevel Optimization

- 通过快照机制让智能体选择性执行昂贵梯度计算,减少计算开销。
- 在合成数据、MNIST、Fashion-MNIST和miniImageNet上均实现高效学习与良好性能。
- 适合需要低通信与计算成本的分布式学习场景,如边缘AI协同训练。
网络化AI系统中,多个智能体需通过通信网络协同学习与自适应模型。此类系统中,超参数优化、数据清洗和元学习常引发双层优化问题,但重复评估梯度、雅可比和海森矩阵会带来巨大计算负担。为此,我们提出快照单循环去中心化双层优化算法(S³LDBO),通过快照机制使智能体间歇跳过高成本的导数计算。该机制可视为网络化AI的自主计算-适应策略:智能体选择性执行耗时本地更新,同时保持全局协同学习。我们在确定性环境下建立了算法的遍历迭代复杂度及高概率非遍历迭代复杂度。实验在合成数据、MNIST上的超参数优化,Fashion-MNIST上的数据超清洗,以及miniImageNet上的去中心化元学习任务中验证了该算法在保持竞争性学习性能的同时显著提升计算效率。
原文摘要 · Abstract (English)
Networked AI systems increasingly rely on multiple agents that collaboratively learn and adapt models over communication networks. In such systems, bilevel formulations naturally arise in hyperparameter optimization, data cleaning, and meta-learning, but the repeated evaluation of gradients, Jacobians, and Hessians can impose a substantial computational burden on individual agents. To address this challenge, we propose Snapshot-SLDBO (S$^3$LDBO), an efficient single-loop decentralized bilevel optimization algorithm that enables agents to intermittently skip expensive derivative evaluations through a snapshot mechanism. This mechanism can be interpreted as an autonomous computation-adaptation strategy for networked AI, where agents selectively perform costly local updates while maintaining global collaborative learning. We establish the ergodic iteration complexity and the high probability nonergodic iteration complexity of the proposed algorithm within a deterministic setting. Experimental results on hyperparameter optimization with synthetic and MNIST datasets, data hyper-cleaning on Fashion-MNIST, and decentralized meta-learning on miniImageNet demonstrate that the proposed algorithm improves computational efficiency while maintaining competitive learning performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。