arXiv:2409.04919cs.LGstat.ML2024-09

提出共享表示学习的统计理论,揭示协作何时真正提升模型性能。

Learning with Shared Representations: Statistical Rates and Efficient Algorithms

  • 设计谱估计器,利用局部均值副本逼近非凸解
  • 发现两种最优率阶段:参数计数与惩罚型,取决于客户端数量和数据量
  • 适用于小样本、异构数据场景,对迁移学习和私有微调有指导意义

通过潜在共享特征表示实现协同学习,使异构客户端能够训练出性能更优、样本复杂度更低的个性化模型。尽管已有广泛应用和大量研究,但此类方法的理论理解仍不完整,即使在低维线性子空间限制下也如此。本文建立了低维共享表示学习的统计误差上界与下界。分析同时涵盖统计异质性(包括协变量和概念漂移)及本地数据集大小差异,这些因素常被以往工作忽略。我们进一步将结果扩展至非线性模型,包括逻辑回归和单隐藏层ReLU网络。具体地,设计了一种谱估计器,利用局部平均的独立副本近似非凸最小二乘解,并推导出近乎匹配的极小极大下界。当共享表示在各客户端间充分覆盖时——即无方向严重欠采样——该估计器达到最优统计速率。结果揭示了最优率的两个不同阶段:标准参数计数阶段与大客户端数或小本地数据量下的惩罚阶段。这些发现精确刻画了协作在迁移学习与私有微调中对整体系统或个体客户端是否带来增益的条件。

原文摘要 · Abstract (English)

Collaborative learning through latent shared feature representations enables heterogeneous clients to train personalized models with improved performance and reduced sample complexity. Despite empirical success and extensive study, the theoretical understanding of such methods remains incomplete, even for representations restricted to low-dimensional linear subspaces. In this work, we establish new upper and lower bounds on the statistical error in learning low-dimensional shared representations across clients. Our analysis captures both statistical heterogeneity (including covariate and concept shifts) and variation in local dataset sizes, aspects often overlooked in prior work. We further extend these results to nonlinear models including logistic regression and one-hidden-layer ReLU networks. Specifically, we design a spectral estimator that leverages independent replicas of local averages to approximate the non-convex least-squares solution and derive a nearly matching minimax lower bound. Our estimator achieves the optimal statistical rate when the shared representation is well covered across clients -- i.e., when no direction is severely underrepresented. Our results reveal two distinct phases of the optimal rate: a standard parameter-counting regime and a penalized regime when the number of clients is large or local datasets are small. These findings precisely characterize when collaboration benefits the overall system or individual clients in transfer learning and private fine-tuning.

协同学习共享表示统计理论迁移学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。