针对语音模型联邦微调中的硬件差异与效率问题，提出可动态终止计算的自适应框架。
Adaptive Federated Fine-Tuning of Self-Supervised Speech Representations
- 在预训练语音模型中间层插入轻量预测头，支持按需提前退出计算
- 采用分层深度感知聚合策略，提升不同网络深度表示的利用效率
- 显著降低边缘设备负担，兼容异构硬件，在资源受限环境仍保持良好性能
将联邦学习（FL）与自监督学习（SSL）结合，可在保护隐私的前提下实现语音任务的微调。然而，联邦环境存在显著异构性：客户端计算能力不同，统一微调易引发慢节点问题；下游任务需求各异，全模型更新效率低下。为此，本文提出一种带早期退出机制的自适应联邦微调框架。在SSL骨干网络中间层插入轻量预测头，使客户端可根据本地约束和任务需求提前终止计算。进一步设计分层、深度感知的部分聚合策略，更高效利用不同网络深度的表示。实验表明，该框架有效降低边缘端开销，支持异构硬件，在资源受限的联邦环境中仍保持竞争力。
原文摘要 · Abstract (English)
Integrating Federated Learning (FL) with self-supervised learning (SSL) enables privacy-preserving fine-tuning for speech tasks. However, federated environments exhibit significant heterogeneity: clients differ in computational capacity, causing straggler effects under unified fine-tuning, while diverse downstream tasks require different representation depths, making full-model updates inefficient. To address these challenges, we propose an adaptive federated fine-tuning framework with early exits. Lightweight prediction heads are inserted at intermediate layers of the SSL backbone, allowing clients to terminate computation based on local constraints and task requirements. We further introduce a layer-wise, depth-aware partial aggregation strategy to better utilize representations from different network depths. Experiments show that the framework reduces edge overhead, supports heterogeneous hardware, and maintains competitive performance in resource-constrained federated environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。