arXiv:2603.07027cs.LG2026-03中稿 · ICLR

让不同算力的客户端共同生成隐私保护的文本数据,提升联邦生成效果。

Resource-Adaptive Federated Text Generation with Differential Privacy

  • 强客户端做私密微调,弱客户端用轻量投票优化合成文本
  • 单轮通信完成弱端参与,合成数据与全局分布对齐度提升27%
  • 适合算力不均、需隐私保护的跨组织文本生成场景

在跨孤岛联邦学习中,敏感文本数据因隐私法规保留在本地组织,反复为下游任务训练既耗通信又难保隐私。一种可行方案是生成差分隐私(DP)合成数据,近似全局分布并可跨任务复用。但预训练大模型常受领域偏移影响,联邦微调又受限于计算异质性:仅资源充足的客户端能更新模型,弱客户端被排除,加剧数据偏差与DP噪声的负面影响。本文提出灵活参与框架,适配客户端能力:强客户端执行DP联邦微调,弱客户端通过轻量级DP投票机制贡献合成文本优化。为确保合成数据反映全局数据分布,引入控制码(如标签、主题、元数据)表示各客户端数据比例,并约束投票至语义一致子集。该两阶段方法仅需一轮通信即可完成弱客户端参与,整合所有参与者贡献。实验表明,本框架显著提升分布对齐性与下游任务鲁棒性,在差分隐私和异构环境下表现更优。

原文摘要 · Abstract (English)

In cross-silo federated learning (FL), sensitive text datasets remain confined to local organizations due to privacy regulations, making repeated training for each downstream task both communication-intensive and privacy-demanding. A promising alternative is to generate differentially private (DP) synthetic datasets that approximate the global distribution and can be reused across tasks. However, pretrained large language models (LLMs) often fail under domain shift, and federated finetuning is hindered by computational heterogeneity: only resource-rich clients can update the model, while weaker clients are excluded, amplifying data skew and the adverse effects of DP noise. We propose a flexible participation framework that adapts to client capacities. Strong clients perform DP federated finetuning, while weak clients contribute through a lightweight DP voting mechanism that refines synthetic text. To ensure the synthetic data mirrors the global dataset, we apply control codes (e.g., labels, topics, metadata) that represent each client's data proportions and constrain voting to semantically coherent subsets. This two-phase approach requires only a single round of communication for weak clients and integrates contributions from all participants. Experiments show that our framework improves distribution alignment and downstream robustness under DP and heterogeneity.

联邦学习隐私生成文本生成差分隐私

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。