首个针对联邦学习中噪声标签的综合评测基准，揭示噪声对模型性能的影响。
FNBench: Benchmarking Robust Federated Learning against Noisy Labels
- 构建FNBench基准，统一评估18种抗噪方法在5个图像和1个文本数据集上的表现。
- 发现三类噪声模式（合成、人工标注、系统性）均显著降低联邦学习精度，最大降幅达25%。
- 提出基于表征感知的正则化方法，可提升现有方法在噪声环境下的鲁棒性。
联邦学习中的标签噪声鲁棒性是重大挑战。从数据视角看，各客户端的数据标注质量难以保证，存在不同程度的复杂噪声，导致模型性能下降。已有研究尝试解决此问题，但缺乏在统一设置下全面评估其实际效果的基准。为此，我们提出了首个基准研究FNBench，涵盖合成噪声、不完美人工标注误差和系统性错误三种典型噪声模式。评估覆盖18种先进方法，涉及5个图像识别数据集和1个文本分类数据集。我们分析了噪声影响联邦学习的原因，并基于观察提出一种表征感知正则化方法，以增强现有方法对噪声的鲁棒性。最后讨论了当前工作的局限，并提出三个未来方向。代码已开源，供社区使用。
原文摘要 · Abstract (English)
Robustness to label noise within data is a significant challenge in federated learning (FL). From the data-centric perspective, the data quality of distributed datasets can not be guaranteed since annotations of different clients contain complicated label noise of varying degrees, which causes the performance degradation. There have been some early attempts to tackle noisy labels in FL. However, there exists a lack of benchmark studies on comprehensively evaluating their practical performance under unified settings. To this end, we propose the first benchmark study FNBench to provide an experimental investigation which considers three diverse label noise patterns covering synthetic label noise, imperfect human-annotation errors and systematic errors. Our evaluation incorporates eighteen state-of-the-art methods over five image recognition datasets and one text classification dataset. Meanwhile, we provide observations to understand why noisy labels impair FL, and additionally exploit a representation-aware regularization method to enhance the robustness of existing methods against noisy labels based on our observations. Finally, we discuss the limitations of this work and propose three-fold future directions. To facilitate related communities, our source code is open-sourced at https://github.com/Sprinter1999/FNBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。