用熵正则化门控机制，提升小样本联邦学习的稀疏模型发现能力。
Entropy-Regularized Probabilistic Gates for Sparse Model Discovery in Scarce-Data Federated Learning
- 引入熵正则化门控，防止过早锁定稀疏结构。
- 在真实与合成数据上，测试性能和稀疏恢复准确率均优于基线方法。
- 适合数据异构、客户端参与不均的小样本联邦学习场景。
联邦学习（FL）是一种分布式机器学习范式，允许多个客户端协作而无需共享数据。在数据异构和部分客户端参与的情况下，学习稀疏模型有助于提升通信与计算效率，但在小样本高维情形（d >> N）下，优化容易导致参数配置无法泛化到未见测试数据。传统基于幅度的剪枝忽略参数空间中的不确定性探索；而采用概率门控与L0约束的方法可在训练中采样多种稀疏配置。本文研究了门控分布的熵正则化机制，以在稀疏联邦优化中保持不确定性，避免过早确定稀疏支持。在合成与真实世界基准上的实验表明，该方法在数据异构、客户端参与异构及不同稀疏度条件下，均持续优于联邦迭代硬阈值（Fed-IHT）和先密集平均后剪枝（FedAvg）的基线方法，在测试数据上的统计性能与稀疏恢复准确率均有提升。
原文摘要 · Abstract (English)
Federated Learning (FL) is a distributed machine learning (ML) paradigm with collaboration among multiple clients without sharing data. FL is challenging under data heterogeneity and partial client participation. Learning sparse models is useful for communication and computational efficiency in FL, but it is especially difficult in the small-sample high-dimensional regime (d >> N) where optimization can yield parameter configurations that fail to generalize to unseen test data. While magnitude-based pruning doesn't account for uncertainty exploration in the parameter space, a formulation with probabilistic gates and an L0 constraint allows sampling from competing sparse configurations during training. In this work, we study entropy regularization of gate distributions as a mechanism to maintain uncertainty in sparse federated optimization by preventing early commitment to sparse support. We examine its impact under data heterogeneity, client participation heterogeneity, and sparsity. Experiments on synthetic and real-world benchmarks show consistent improvements over federated iterative hard thresholding (Fed-IHT) and pruning after dense federated averaging (FedAvg) training, both in statistical performance on test data and in sparsity recovery accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。