将mixup引入拓扑知识蒸馏,提升可穿戴传感器数据的模型性能。
Role of Mixup in Topological Persistence Based Knowledge Distillation for Wearable Sensor Data
- 用多教师知识蒸馏融合时序与拓扑特征,压缩模型体积。
- mixup通过平滑标签增强训练鲁棒性,提升学生模型准确率。
- 适合做轻量化可穿戴设备智能分析的研究者参考。
可穿戴传感器数据分析在多个应用中取得成功。为充分表示高采样率时间序列,拓扑数据分析(TDA)被引入,发现其能补充传统时间序列特征。然而,通过TDA提取拓扑特征耗时长且计算资源消耗大,难以部署。为此,知识蒸馏(KD)技术可用于模型压缩与迁移学习,通过从大模型中迁移知识生成小型学生模型。利用多个教师模型,可同时传递时序与拓扑特征,最终得到仅依赖时序数据的优越学生模型。另一方面,mixup作为流行的数据增强技术,能通过混合样本和标签提升训练鲁棒性。KD与mixup均依赖平滑分布的学习策略:前者从教师模型获取平滑输出分布,后者通过混合标签生成平滑目标。二者共享这一平滑机制,形成连接纽带。本文系统分析了mixup在基于时序与拓扑持久性知识蒸馏中的作用,评估了多种KD与mixup方法在可穿戴传感器数据上的表现。
原文摘要 · Abstract (English)
The analysis of wearable sensor data has enabled many successes in several applications. To represent the high-sampling rate time-series with sufficient detail, the use of topological data analysis (TDA) has been considered, and it is found that TDA can complement other time-series features. Nonetheless, due to the large time consumption and high computational resource requirements of extracting topological features through TDA, it is difficult to deploy topological knowledge in various applications. To tackle this problem, knowledge distillation (KD) can be adopted, which is a technique facilitating model compression and transfer learning to generate a smaller model by transferring knowledge from a larger network. By leveraging multiple teachers in KD, both time-series and topological features can be transferred, and finally, a superior student using only time-series data is distilled. On the other hand, mixup has been popularly used as a robust data augmentation technique to enhance model performance during training. Mixup and KD employ similar learning strategies. In KD, the student model learns from the smoothed distribution generated by the teacher model, while mixup creates smoothed labels by blending two labels. Hence, this common smoothness serves as the connecting link that establishes a connection between these two methods. In this paper, we analyze the role of mixup in KD with time-series as well as topological persistence, employing multiple teachers. We present a comprehensive analysis of various methods in KD and mixup on wearable sensor data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。