大模型在极端数据不均时反而让最差客户更惨，颠覆了预训练有益的普遍认知。
When More Parameters Hurt: Foundation Model Priors Amplify Worst-Client Disparity Under Extreme Federated Heterogeneity
- 对比小模型与大模型在联邦学习中的表现，发现大模型在极端数据不均下更差
- 极端偏斜下大模型最差客户准确率差距达50.1%~56%，远超小模型的32.2%
- 揭示大模型公平性悖论，提醒医疗教育等高风险场景需专门保护少数客户端
联邦学习（FL）正被用于在分布式私有数据上微调基础模型（FMs）。社区普遍认为大规模预训练如同‘潮水托起所有船只’。然而，我们的实验显示，在极端异构条件下，这些强大先验反而会损害最弱势客户端的表现。在联邦文本分类任务中，我们在四个非独立同分布（Non-IID）异构水平下对比了TextCNN（270万参数）和带低秩适配（LoRA，6600万参数）的DistilBERT。在极端标签偏斜（alpha = 0.1）条件下，DistilBERT+LoRA 的最差客户准确率差距为50.1%至56%，比TextCNN的32.2%高出56%；尽管其参数量是25倍且经过充分预训练。在中度异构（alpha >= 0.5）下，模式反转：大模型几乎消除了差距。我们称之为FM公平性悖论。进一步表明，反向加权的LoRA聚合方法（FedAvgW）无法缓解此差距，说明仅靠聚合重加权可能不足。结果强调，部署基础模型于医疗、教育等高风险联邦场景前，必须建立显式保护少数客户端的机制。
原文摘要 · Abstract (English)
Federated learning (FL) is increasingly used to fine-tune foundation models (FMs) on distributed private data. The community largely assumes that large-scale pretraining serves as a 'rising tide that lifts all boats' in federated settings. However, our experiments reveal that these powerful priors can hinder rather than help the most disadvantaged clients under extreme heterogeneity. Through controlled experiments on federated text classification, we compare worst-client accuracy between TextCNN (2.7M parameters) and DistilBERT with Low-Rank Adaptation (LoRA, 66M parameters) across four Non-IID heterogeneity levels. Under extreme label skew (alpha = 0.1), DistilBERT+LoRA produces a worst-client accuracy gap of 50.1% -- 56% larger than TextCNN's 32.2% gap, despite having 25x more parameters and extensive pretraining. Under moderate heterogeneity (alpha >= 0.5), the pattern reverses: the FM nearly eliminates the gap. We call this the FM Fairness Paradox. We further show that an inverse-weighted LoRA aggregation method (FedAvgW) does not resolve the disparity, suggesting aggregation reweighting alone may be insufficient. Our results highlight the need for mechanisms that explicitly protect minority clients before deploying foundation models in high-stakes federated contexts such as healthcare and education.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。