让多个神经网络自动互教互学,持续提升整体性能。
Collaborative Knowledge Distillation via a Learning-by-Education Node Community
- 节点动态扮演教师或学生,自主发现知识源并实时蒸馏。
- 在图像分类中使群体平均准确率稳步提升,有效缓解灾难性遗忘。
- 适合分布式场景下无任务边界信息的持续学习,无需人工干预。
本文提出一种新型协同知识蒸馏框架——学习即教育节点社区(LENC),通过部署的深度神经网络(DNN)节点间动态自主的知识交换,实现持续集体学习。这些DNN节点可自主选择作为学生寻求知识或作为教师传授知识,形成协作学习环境。LENC通过自主教师发现和流式驱动的DNN蒸馏,在无需显式任务边界标记的情况下,支持任务无关的持续适应,并增强学习能力与协作性。该框架有效利用新任务学习时的精选同行教师知识,缓解了灾难性遗忘问题。在概念验证实现的实验中,LENC展现了在多种学习与推理场景下的功能优势,通过合理利用所有节点同伴的集体知识,逐步提升交互式DNN节点群体在图像分类任务中的平均测试准确率。其在线无标签协同知识蒸馏表现优异。
原文摘要 · Abstract (English)
A novel Learning-by-Education Node Community framework (LENC) for Collaborative Knowledge Distillation (CKD) is presented, which facilitates continual collective learning through effective knowledge exchanges among diverse deployed Deep Neural Network (DNN) peer nodes. These DNNs dynamically and autonomously adopt either the role of a student, seeking knowledge, or that of a teacher, imparting knowledge, fostering a collaborative learning environment. The proposed framework triggers knowledge transfer via autonomous teacher discovery and stream-driven DNN distillation as needed, while enhancing their learning capabilities and promoting their collaboration. LENC addresses the challenges of handling diverse training data distributions and the limitations of individual DNN node learning abilities. \hl{It enables the exploitation of selected peer-teacher knowledge upon learning a new task and mitigates catastrophic forgetting in DNN nodes.} \hl{Additionally, it supports task-boundary-free continual adaptation in distributed settings via autonomous role assignment and modular forgetting mitigation, as DNN nodes receive no explicit task-boundary metadata during deployment.} Experimental evaluation on a proof-of-concept implementation demonstrates the LENC framework's functionalities and benefits across multiple DNN learning and inference scenarios. The conducted experiments showcase its ability to gradually improve the average test accuracy of the community of interacting DNN nodes in image classification problems, by appropriately leveraging the collective knowledge of all node peers. The LENC framework achieves strong performance in on-line unlabelled CKD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。