提出FIRE方法，提升分布式训练下的模型验证鲁棒性
Technical note on Fisher Information for Robust Federated Cross-Validation
- 用近似Fisher信息量化数据碎片化带来的分布偏移
- 在偏移验证集上相比基准提升最高5.3%
- 适合处理跨地域联邦学习中的分布不一致问题
当训练数据分散在不同批次或跨地理区域的联邦学习中时，模型性能会下降。这主要源于数据在时空上碎片化引发的协变量偏移，导致各片段的实证训练分布与假设的未碎片化训练分布及单一验证分布存在差异。为此，我们提出稳健联邦验证的费雪信息（FIRE）。该方法通过近似费雪信息累积碎片化引起的协变量偏移差异，并将其作为每个片段的损失惩罚项，实现可扩展的分布对齐。我们证明该估计更易计算，且在偏移验证集上，相比重要性加权基线最高提升5.1%，相比联邦学习基线最高提升5.3%。
原文摘要 · Abstract (English)
When training data are fragmented across batches or federated-learned across different geographic locations, trained models manifest performance degradation. That degradation partly owes to covariate shift induced by data having been fragmented across time and space and producing dissimilar empirical training distributions. Each fragment's distribution is slightly different to a hypothetical unfragmented training distribution of covariates, and to the single validation distribution. To address this problem, we propose Fisher Information for Robust fEderated validation (\textbf{FIRE}). This method accumulates fragmentation-induced covariate shift divergences from the global training distribution via an approximate Fisher information. That term, which we prove to be a more computationally-tractable estimate, is then used as a per-fragment loss penalty, enabling scalable distribution alignment. FIRE outperforms importance weighting benchmarks by $5.1\%$ at maximum and federated learning (FL) benchmarks by up to $5.3\%$ on shifted validation sets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。