通过核方法证明了无需共享数据的协作学习可达到近最优性能。
A Kernel Perspective on Distillation-based Collaborative Learning
- 从核回归视角分析蒸馏协作算法，建立理论框架。
- 在异质分布式环境下实现近极小极大最优性能。
- 提出基于神经网络特征核匹配的实用算法，适合隐私保护场景。
过去十年，协作学习在提升多方AI模型性能方面受到广泛关注。然而，在不共享各参与方私有数据与模型的前提下提升性能仍具挑战。近年来，基于蒸馏的算法利用公开无标签数据成为新方向，但其理论与实践效果仍不理想。本文从核回归视角，首次严格分析代表性蒸馏算法，证明了在大规模分布式、统计异质环境中，不直接共享本地数据或模型的非参数化协作学习算法具有（近）极小极大最优性。基于此理论结果，我们提出一种基于神经网络架构的实用蒸馏协作学习算法，通过特征核匹配有效连接理论假设与实际神经网络设置。我们在多种回归任务上进行仿真，验证了理论正确性，并展示了所提算法的可行性。
原文摘要 · Abstract (English)
Over the past decade, there is a growing interest in collaborative learning that can enhance AI models of multiple parties. However, it is still challenging to enhance performance them without sharing private data and models from individual parties. One recent promising approach is to develop distillation-based algorithms that exploit unlabeled public data but the results are still unsatisfactory in both theory and practice. To tackle this problem, we rigorously analyze a representative distillation-based algorithm in the view of kernel regression. This work provides the first theoretical results to prove the (nearly) minimax optimality of the nonparametric collaborative learning algorithm that does not directly share local data or models in massively distributed statistically heterogeneous environments. Inspired by our theoretical results, we also propose a practical distillation-based collaborative learning algorithm based on neural network architecture. Our algorithm successfully bridges the gap between our theoretical assumptions and practical settings with neural networks through feature kernel matching. We simulate various regression tasks to verify our theory and demonstrate the practical feasibility of our proposed algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。