通过知识蒸馏与动态量化,让多模态音频视觉模型在边缘设备上高效运行。
Efficient Audio-Visual Event Recognition via Knowledge Distillation and Dynamic INT8 Quantization of a Hybrid Cross-Attention Network

- 用教师-学生框架压缩混合注意力网络,降低参数量和计算开销。
- 在AVE数据集上参数减少59.06%,准确率仅下降2.14%。
- 动态INT8量化使模型体积从10.71MB降至2.04MB,适合边缘部署。
基于Transformer的多模态音频-视觉事件识别(AVER)虽性能优异,但计算复杂度高、内存占用大,难以在边缘设备部署。本文提出一种结合架构压缩、知识蒸馏与动态INT8量化的方法,构建高效压缩框架。采用高容量教师模型,融合VideoMAE视觉表示、AST音频特征提取及混合交叉注意力融合网络。轻量级学生模型通过减小隐藏维度、注意力头数与前馈网络规模实现压缩,再通过知识蒸馏迁移教师模型判别性知识。最后应用动态INT8后训练量化进一步缩减模型体积。在AVE数据集上的实验表明,该框架使多模态融合模块可训练参数减少59.06%,分类准确率仅下降2.14%;动态INT8量化将模型大小从10.71MB降至2.04MB,同时保持良好识别性能。结果证明该方法在精度与效率间取得良好平衡,适用于资源受限的边缘AI平台。
原文摘要 · Abstract (English)
Audio-visual event recognition (AVER) has achieved significant performance improvements through transformer-based multimodal architectures. However, the high computational complexity, large memory footprint, and inference cost of these models hinder their deployment on edge and resource-constrained devices. This paper presents an efficient compression framework for hybrid cross-attention-based audiovisual event recognition by combining architectural model compression, knowledge distillation, and dynamic INT8 quantization. A high-capacity teacher model integrates VideoMAE for visual representation learning, the Audio Spectrogram Transformer (AST) for audio feature extraction, and a hybrid cross-attention fusion network for multimodal feature integration. A lightweight student model is constructed by reducing the hidden feature dimension, the number of attention heads, and the feedforward network size while preserving the overall network architecture. The student model is trained using knowledge distillation to effectively transfer discriminative knowledge from the teacher. Finally, dynamic INT8 post-training quantization is applied to further reduce the model size for efficient deployment. Experimental results on the Audio-Visual Event (AVE) dataset show that the proposed framework reduces the number of trainable parameters in the multimodal fusion module by 59.06%, with only a 2.14% decrease in classification accuracy compared with the teacher model. Furthermore, dynamic INT8 quantization reduces the model size from 10.71 MB to 2.04 MB while maintaining competitive recognition performance. These results demonstrate that the proposed framework provides an effective trade-off between recognition accuracy and computational efficiency, making it a promising solution for deployment on resource-constrained edge AI platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。