用压缩技术让生成模型安全检测器在嵌入式设备上实时运行
Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment
- 结合量化、剪枝和知识蒸馏压缩变分自编码器
- 推理速度提升20%-28%，内存大幅减少，性能损失小于5%
- 适合资源受限的嵌入式系统安全监控场景
分布外（OOD）检测器可在嵌入式网络物理系统中作为安全监控，识别超出模型训练分布的样本以避免潜在危险操作。然而，现有方法多采用深度神经网络实现，难以满足嵌入式系统在内存与功耗限制下的实时性要求。本文聚焦于基于变分自编码器（VAE）的OOD检测器，其在隐空间中完成检测任务，综合应用量化、剪枝与知识蒸馏技术。尽管这些压缩手段会略微增加测试损失，但对隐空间中的OOD检测性能影响较小，因此可有效构建轻量级检测器，实现在嵌入式CPU与GPU上的实时推理。本文提出一套融合三种压缩技术的设计方法，在保持原有检测器AUROC性能的前提下，显著降低内存占用与推理时间。实验基于Jetson Nano平台，对两个现有检测器进行验证，使GPU与CPU推理时间分别减少20%和28%，同时维持AUROC在基准值的5%以内。
原文摘要 · Abstract (English)
Out-of-distribution (OOD) detectors can act as safety monitors in embedded cyber-physical systems by identifying samples outside a machine learning model's training distribution to prevent potentially unsafe actions. However, OOD detectors are often implemented using deep neural networks, which makes it difficult to meet real-time deadlines on embedded systems with memory and power constraints. We consider the class of variational autoencoder (VAE) based OOD detectors where OOD detection is performed in latent space, and apply quantization, pruning, and knowledge distillation. These techniques have been explored for other deep models, but no work has considered their combined effect on latent space OOD detection. While these techniques increase the VAE's test loss, this does not correspond to a proportional decrease in OOD detection performance and we leverage this to develop lean OOD detectors capable of real-time inference on embedded CPUs and GPUs. We propose a design methodology that combines all three compression techniques and yields a significant decrease in memory and execution time while maintaining AUROC for a given OOD detector. We demonstrate this methodology with two existing OOD detectors on a Jetson Nano and reduce GPU and CPU inference time by 20% and 28% respectively while keeping AUROC within 5% of the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。