大模型与小模型协同，解决联邦持续学习中的遗忘与数据孤岛问题。
Large-Small Model Collaborative Framework for Federated Continual Learning
- 小模型作为动态桥梁，在本地持续适应新任务。
- 在多个客户端上实现优于单一大模型的性能提升。
- 适合资源受限但需持续学习的分布式场景。
基础模型（FMs）在联邦持续学习（FCL）中面临严峻挑战：各客户端在严格的数据与通信约束下，需从私有的、动态演化的任务流中学习。尽管基础模型具备强大的泛化能力，却难以利用本地私有数据，导致下游任务表现不佳。同时，由于参数量巨大、模型复杂度高，避免遗忘旧知识尤为困难。相比之下，小模型可在资源受限条件下本地训练，并受益于成熟的持续学习技术。为此，我们提出首个联邦持续学习中的协同框架，让轻量级本地模型作为动态桥梁，持续适应新任务，从而增强大模型的实用性。引入两个新组件：小模型持续微调以防止时间遗忘；逐次蒸馏用于在服务器端个性化融合异构本地知识。实验表明，即使客户端使用异构小模型，该框架仍表现出色。
原文摘要 · Abstract (English)
Continual learning (CL) for Foundation Models (FMs) is an essential yet underexplored challenge, especially in Federated Continual Learning (FCL), where each client learns from a private, evolving task stream under strict data and communication constraints. Despite their powerful generalization abilities, FMs often exhibit suboptimal performance on local downstream tasks, as they are unable to utilize private local data. Furthermore, enabling FMs to learn new tasks without forgetting prior knowledge is inherently a challenging problem, primarily due to their immense parameter count and high model complexity. In contrast, small models can be trained locally under resource-constrained conditions and benefit from more mature CL techniques. To bridge the gap between small models and FMs, we propose the first collaborative framework in FCL, where lightweight local models act as a dynamic bridge, continually adapting to new tasks while enhancing the utility of the large model. Two novel components are also included: Small Model Continual Fine-tuning is for preventing small models from temporal forgetting; One-by-One Distillation performs personalized fusion of heterogeneous local knowledge on the server. Experimental results demonstrate its superior performance, even when clients utilize heterogeneous small models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。