联邦学习中低秩矩阵分解的高效收敛分析,提升计算效率与精度。
In-depth Analysis of Low-rank Matrix Factorisation in a Federated Setting
- 通过幂初始化将非凸问题转为强凸,用并行Nesterov梯度法求解
- 收敛速度依赖σ_max/σ_r,优于文献中σ_max²/σ_min²的旧结果
- 适用于多方协作的数据建模,尤其适合高维稀疏场景
我们分析了一种分布式算法,用于在N个客户端上计算低秩矩阵分解,每个客户端持有本地数据集$[1m[31m\mathbf{S}^i ∈ \mathbb{R}^{n_i \times d}[0m$。目标是最小化$[1m[31m\frac{1}{2} \sum_{i=1}^N \|\mathbf{S}^i - \mathbf{U}^i \mathbf{V}^\top\|^2_{\text{F}}[0m$,其中$[1m[31m\mathbf{U}^i ∈ \mathbb{R}^{n_i\times r}[0m$为本地变量,$[1m[31m\mathbf{V} ∈ \mathbb{R}^{d \times r}[0m$为全局共享变量。在幂初始化下,原光滑非凸问题被重写为光滑强凸问题,采用并行Nesterov梯度下降法,仅需初始通信一步。理论分析表明,过失损失以线性速率收敛,其依赖于$[1m[31m\sigma_{\max} / \sigma_r[0m$,其中$[1m[31m\sigma_r[0m$是所有$[1m[31m\mathbf{S}^i[0m$拼接矩阵$[1m[31m\mathbf{S}[0m$的第$r$个奇异值,优于现有依赖$[1m[31m\sigma_{\max}^2 / \sigma_{\min}^2[0m$的收敛率。我们还给出了重构误差的Frobenius范数上界,并在合成与真实数据上完成实验验证。
原文摘要 · Abstract (English)
We analyze a distributed algorithm to compute a low-rank matrix factorization on $N$ clients, each holding a local dataset $\mathbf{S}^i \in \mathbb{R}^{n_i \times d}$, mathematically, we seek to solve $min_{\mathbf{U}^i \in \mathbb{R}^{n_i\times r}, \mathbf{V}\in \mathbb{R}^{d \times r} } \frac{1}{2} \sum_{i=1}^N \|\mathbf{S}^i - \mathbf{U}^i \mathbf{V}^\top\|^2_{\text{F}}$. Considering a power initialization of $\mathbf{V}$, we rewrite the previous smooth non-convex problem into a smooth strongly-convex problem that we solve using a parallel Nesterov gradient descent potentially requiring a single step of communication at the initialization step. For any client $i$ in $\{1, \dots, N\}$, we obtain a global $\mathbf{V}$ in $\mathbb{R}^{d \times r}$ common to all clients and a local variable $\mathbf{U}^i$ in $\mathbb{R}^{n_i \times r}$. We provide a linear rate of convergence of the excess loss which depends on $σ_{\max} / σ_{r}$, where $σ_{r}$ is the $r^{\mathrm{th}}$ singular value of the concatenation $\mathbf{S}$ of the matrices $(\mathbf{S}^i)_{i=1}^N$. This result improves the rates of convergence given in the literature, which depend on $σ_{\max}^2 / σ_{\min}^2$. We provide an upper bound on the Frobenius-norm error of reconstruction under the power initialization strategy. We complete our analysis with experiments on both synthetic and real data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。