解决联邦学习中低质数据干扰问题，提升大模型指令微调效果
Data Quality Control in Federated Instruction-tuning of Large Language Models
- 客户端用轻量推理评估数据质量，自动识别优质问答对
- 按质量从高到低分层训练，逐步引入低质量数据提升学习效率
- 支持动态调整，适合数据分布不均的隐私保护场景
联邦学习（FL）通过利用海量分布式数据实现大语言模型（LLM）的隐私保护协同指令微调。然而，由于客户端缺乏全局视角，难以在训练前过滤噪声或低质量样本，导致数据质量问题加剧。为此，我们提出FedDQC，一种具备动态数据质量控制的新型联邦指令微调框架。该方法引入两项关键创新：首先，提出指令-响应对齐（IRA）机制，一种仅需低成本推理的客户端侧质量评估指标，验证了高IRA数据对应更相关、更易学习的问答对；其次，模仿人类由易到难的知识获取过程，设计质量感知的分层联邦训练框架，模型在协作下从高至低IRA数据逐级微调。框架还支持每层自适应质量评估，实现训练过程中的动态调整。在合成与真实数据集上的大量实验表明，该方法显著提升了联邦学习中混合质量数据下的大模型性能。
原文摘要 · Abstract (English)
Federated Learning (FL) enables privacy-preserving collaborative instruction tuning of large language models (LLMs) by leveraging massively distributed data. However, the decentralized nature of FL exacerbates data quality challenges, as local clients lack global visibility to filter noisy or low-quality samples before training. To resolve this issue, we propose FedDQC, a novel federated instruction tuning framework with dynamic data quality control. Our approach introduces two key innovations. First, we propose instruction-response alignment (IRA), an efficient client-side metric for quality evaluation requiring only low-cost inference. We validate that higher-IRA data corresponds to more relevant and easier-to-learn question-answer pairs. Second, mirroring the human easy-to-hard knowledge acquisition process, we design a quality-aware hierarchical FL training framework, where the LLM is progressively fine-tuned from high- to low-IRA data in a collaborative manner. The framework also supports adaptive data quality assessment at each hierarchy, enabling dynamic adjustments throughout the training process. Extensive experiments on synthetic and real-world datasets show that our method significantly improves LLM performance on mixed-quality data in FL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。