arXiv:2410.17986cs.LGcs.AI2024-10NeurIPS被引 16

提出FeT框架,解决多方模糊数据下的联邦学习难题

Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data

  • 将模糊标识编码进数据表示,用分布式Transformer建模
  • 支持50方协作,准确率最高提升46%,隐私保护成本更低
  • 适合跨机构、数据难对齐的联合建模场景

联邦学习(FL)允许多方在不共享原始数据的情况下协同训练模型。垂直联邦学习(VFL)在跨组织合作中尤为重要,各方贡献同一组实例的不同特征。实际场景中,各方常通过模糊标识关联,形成多方向模糊VFL。现有方法通常只处理多方或双方面模糊情形,扩展到真实多方向模糊场景时性能下降明显且隐私成本高。为此,本文提出联邦Transformer(FeT)框架,支持多方模糊标识下的VFL。FeT创新性地将模糊标识编码为数据表示,并采用分布式的Transformer架构,结合三项新技术提升性能。此外,构建了融合差分隐私与安全多方计算的多方向隐私框架,在保护本地表示的同时最小化效用损失。实验表明,当扩展至50方时,FeT相比基线模型准确率最高提升46%;在双方面模糊设置下,也优于前沿模型,在性能与隐私间取得更好平衡。

原文摘要 · Abstract (English)

Federated Learning (FL) is an evolving paradigm that enables multiple parties to collaboratively train models without sharing raw data. Among its variants, Vertical Federated Learning (VFL) is particularly relevant in real-world, cross-organizational collaborations, where distinct features of a shared instance group are contributed by different parties. In these scenarios, parties are often linked using fuzzy identifiers, leading to a common practice termed as multi-party fuzzy VFL. Existing models generally address either multi-party VFL or fuzzy VFL between two parties. Extending these models to practical multi-party fuzzy VFL typically results in significant performance degradation and increased costs for maintaining privacy. To overcome these limitations, we introduce the Federated Transformer (FeT), a novel framework that supports multi-party VFL with fuzzy identifiers. FeT innovatively encodes these identifiers into data representations and employs a transformer architecture distributed across different parties, incorporating three new techniques to enhance performance. Furthermore, we have developed a multi-party privacy framework for VFL that integrates differential privacy with secure multi-party computation, effectively protecting local representations while minimizing associated utility costs. Our experiments demonstrate that the FeT surpasses the baseline models by up to 46\% in terms of accuracy when scaled to 50 parties. Additionally, in two-party fuzzy VFL settings, FeT also shows improved performance and privacy over cutting-edge VFL models.

联邦学习模糊匹配Transformer隐私保护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。