模仿人类学习过程，分阶段高效微调大模型。
Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stages
- 分阶段逐步构建模型，每阶段优化带宽递增的子模型。
- 相比顶尖方法，训练速度提升4.59倍，通信量减少10.67倍。
- 适合资源受限设备部署，且兼容现有联邦学习框架。
联邦微调使大语言模型在保护数据隐私的前提下适应下游任务，但其高资源消耗限制了边缘设备上的应用。本文提出认知发展式联邦微调（DevFT），受人类认知发展启发，从紧凑基础模型逐步构建强大语言模型。该方法将微调过程分解为多个发展阶段，每个阶段优化参数容量递增的子模型，并将前期知识迁移至后续子模型，提供优化初始化参数，避免陷入局部最优并加速训练。这一范式模拟人类学习过程，渐进构建完整知识体系并精炼已有技能。为高效构建各阶段子模型，DevFT引入去冲突引导的层分组与基于差分的层融合，以提炼关键信息并构建代表性层。在多个基准测试中，DevFT显著优于当前最佳方法：收敛速度最高提升4.59倍，通信开销降低10.67倍，平均性能提升9.07%，同时保持与现有方法的兼容性。
原文摘要 · Abstract (English)
Federated fine-tuning enables Large Language Models (LLMs) to adapt to downstream tasks while preserving data privacy, but its resource-intensive nature limits deployment on edge devices. In this paper, we introduce Developmental Federated Tuning (DevFT), a resource-efficient approach inspired by cognitive development that progressively builds a powerful LLM from a compact foundation. DevFT decomposes the fine-tuning process into developmental stages, each optimizing submodels with increasing parameter capacity. Knowledge from earlier stages transfers to subsequent submodels, providing optimized initialization parameters that prevent convergence to local minima and accelerate training. This paradigm mirrors human learning, gradually constructing comprehensive knowledge structure while refining existing skills. To efficiently build stage-specific submodels, DevFT introduces deconfliction-guided layer grouping and differential-based layer fusion to distill essential information and construct representative layers. Evaluations across multiple benchmarks demonstrate that DevFT significantly outperforms state-of-the-art methods, achieving up to 4.59$\times$ faster convergence, 10.67$\times$ reduction in communication overhead, and 9.07% average performance improvement, while maintaining compatibility with existing approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。