通过离线记录与负权自蒸馏,实现高效点云分类模型压缩。
Efficient Point Cloud Classification via Offline Distillation Framework and Negative-Weight Self-Distillation Technique
- 离线记录教师模型输出,避免师生模型同时加载
- 使用形状级增强减少数据量,提升训练效率
- 负权自蒸馏防止学生模型过拟合,提升性能
点云处理技术的快速发展催生了对高效紧凑模型的迫切需求。知识蒸馏作为模型压缩的有效手段,但传统方法需频繁调用大模型前向推理,显著降低学生模型训练效率并增加资源消耗。为此,本文提出一种创新的离线记录策略:将大量增强样本输入教师模型,仅记录数据增强参数与对应logit输出,避免师生模型同时加载,大幅降低硬件需求。通过采用随机缩放、平移等形状级增强,排除点级抖动等操作,有效缩小记录规模。为缓解小模型过度模仿教师输出导致陷入次优解的问题,引入负权自蒸馏机制。实验表明,该方法在参数量更低的前提下,达到与当前最优模型相当的分类性能,实现了性能与复杂度间的良好平衡。该研究展示了该方法在资源受限环境下优化点云分类任务知识蒸馏的潜力。
原文摘要 · Abstract (English)
The rapid advancement in point cloud processing technologies has significantly increased the demand for efficient and compact models that achieve high-accuracy classification. Knowledge distillation has emerged as a potent model compression technique. However, traditional KD often requires extensive computational resources for forward inference of large teacher models, thereby reducing training efficiency for student models and increasing resource demands. To address these challenges, we introduce an innovative offline recording strategy that avoids the simultaneous loading of both teacher and student models, thereby reducing hardware demands. This approach feeds a multitude of augmented samples into the teacher model, recording both the data augmentation parameters and the corresponding logit outputs. By applying shape-level augmentation operations such as random scaling and translation, while excluding point-level operations like random jittering, the size of the records is significantly reduced. Additionally, to mitigate the issue of small student model over-imitating the teacher model's outputs and converging to suboptimal solutions, we incorporate a negative-weight self-distillation strategy. Experimental results demonstrate that the proposed distillation strategy enables the student model to achieve performance comparable to state-of-the-art models while maintaining lower parameter count. This approach strikes an optimal balance between performance and complexity. This study highlights the potential of our method to optimize knowledge distillation for point cloud classification tasks, particularly in resource-constrained environments, providing a novel solution for efficient point cloud analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。