通过正反知识蒸馏,实现边缘模型安全更新而不泄露用户数据。
DiReDi: Distillation and Reverse Distillation for AIoT Applications
- 先用云端大模型蒸馏出边缘模型,再通过反向蒸馏提取用户差异知识
- 仅上传知识差异而非原始数据,保护隐私且减少冗余信息
- 适合需要持续优化边缘模型又严守数据隐私的AIoT场景
在实际应用中,通常通过部署不同边缘AI模型实现高效运行,由云端大模型远程管理。然而,为特定用户定制或扩展边缘模型至新场景仍具挑战性。用户不当的本地训练或微调可能导致模型失效,甚至引发制造商的法律风险。为此,本文提出创新框架DiReDi,包含知识蒸馏(KD)与反向知识蒸馏(RD)。首先,基于预设数据和云端大模型,在云服务器上对边缘模型进行训练并完成知识蒸馏;随后将仅用于推理的边缘模型分发至用户端。当需根据实际场景更新模型时,利用用户专属数据执行反向蒸馏,从边缘模型中提取用户偏好与厂商预设之间的知识差异,仅将该差异回传至云端以更新云模型,从而保障用户隐私。更新后的云端模型可进一步向边缘模型注入新知识。仿真结果表明,该框架可在不暴露私有数据的前提下,让制造商基于真实场景学习新知识,实现边缘模型的安全有效更新,同时降低初始冗余知识。
原文摘要 · Abstract (English)
Typically, the significant efficiency can be achieved by deploying different edge AI models in various real world scenarios while a few large models manage those edge AI models remotely from cloud servers. However, customizing edge AI models for each user's specific application or extending current models to new application scenarios remains a challenge. Inappropriate local training or fine tuning of edge AI models by users can lead to model malfunction, potentially resulting in legal issues for the manufacturer. To address aforementioned issues, this paper proposes an innovative framework called "DiReD", which involves knowledge DIstillation & REverse DIstillation. In the initial step, an edge AI model is trained with presumed data and a KD process using the cloud AI model in the upper management cloud server. This edge AI model is then dispatched to edge AI devices solely for inference in the user's application scenario. When the user needs to update the edge AI model to better fit the actual scenario, the reverse distillation (RD) process is employed to extract the knowledge: the difference between user preferences and the manufacturer's presumptions from the edge AI model using the user's exclusive data. Only the extracted knowledge is reported back to the upper management cloud server to update the cloud AI model, thus protecting user privacy by not using any exclusive data. The updated cloud AI can then update the edge AI model with the extended knowledge. Simulation results demonstrate that the proposed "DiReDi" framework allows the manufacturer to update the user model by learning new knowledge from the user's actual scenario with private data. The initial redundant knowledge is reduced since the retraining emphasizes user private data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。