arXiv:2607.03763cs.LGcs.AI2026-07

提出FedACT,通过坐标级信任调节提升异构数据下Transformer的联邦训练鲁棒性。

FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity

  • 基于全局修正方向,按坐标信任度重新分配更新幅度。
  • 在强异构数据下,对Vision Transformer提升显著,最高增益达12.3%。
  • 适合需要高鲁棒性的联邦学习场景,如医疗、金融等异构数据应用。

联邦Transformer训练越来越多依赖本地AdamW,其自适应更新可提供远超SGD的本地进展。但在客户端数据异构情况下,即使经过全局修正的AdamW更新,在坐标层面仍存在显著可靠性差异,这种现象称为坐标信任不匹配。现有联邦自适应优化器主要在客户端更新或通信轮次层面解决此问题,但对修正后的自适应方向仍进行密集且均匀的坐标更新。本文提出FedACT,一种面向联邦AdamW训练的全局感知坐标信任调制方法。FedACT首先形成全局修正的自适应方向,再根据坐标级信任分数重新分配更新幅度:对本地梯度与全局修正共同支持的坐标赋予更大步长,同时保留其余坐标的较小非零更新。在联邦视觉Transformer、CNN、大模型预训练及微调任务上的大量实验表明,FedACT持续优于强基线,尤其在强数据异构下的Transformer模型上取得最大提升。机制分析进一步显示,FedACT提升了跨客户端方向一致性,表明坐标级信任分配有效补充了轮次级全局-局部修正。代码将开源。

原文摘要 · Abstract (English)

Federated Transformer training increasingly relies on local AdamW, whose adaptive updates can provide much stronger local progress than SGD-based training. However, under heterogeneous client data, even globally corrected AdamW updates may remain highly uneven in coordinate-wise reliability. We refer to this phenomenon as coordinate trust mismatch. Existing federated adaptive optimizers mainly address mismatch at the client-update or communication-round level, but still apply the corrected adaptive direction densely and uniformly across coordinates. In this paper, we propose FedACT, a global-aware coordinate trust modulation method for federated AdamW training. FedACT first forms a globally corrected adaptive direction and then reallocates update magnitudes according to a coordinate-wise trust score, assigning larger steps to coordinates jointly supported by local gradients and global correction, while preserving smaller non-zero updates on the remaining coordinates. Extensive experiments on federated vision Transformers, CNNs, LLM pre-training, and LLM fine-tuning show that FedACT consistently improves over strong federated adaptive baselines, with the largest gains on Transformer models under stronger data heterogeneity. Mechanism analyses further show that FedACT improves cross-client direction consistency, suggesting that coordinate-level trust allocation effectively complements round-level global-local correction. Code will be released.

联邦学习自适应优化异构数据Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。