用蒸馏技术让小模型实现大模型的多模态表现
DAVSS: Distilled Audio-Visual State Space Models
- 将变压器蒸馏到音视频状态空间模型，提升效率
- 1400万参数仅是Transformer的1/12，性能仍更优
- 更深的跨模态融合，计算成本几乎不变
将基于Transformer的教师模型知识蒸馏到状态空间模型（SSM），在保持高效的同时提升性能。本文提出音视频状态空间模型DAVSS，仅1400万参数，是CAV-MAE等基于Transformer模型的1/12，但性能仍更优。其优势在于：1）采用更小的图像块尺寸，提高输入分辨率，通过增加序列长度弥补模型规模限制，实验证明更大块尺寸会降低性能；2）将30%的模型容量用于联合音视频建模，远超CAV-MAE中不足5%的水平，实现更深层次的跨模态交互，且不显著增加因拼接音视频令牌带来的计算开销。
原文摘要 · Abstract (English)
State-space models (SSMs) distilled from transformer teachers combine the performance of transformers with the efficiency of SSMs. We extend the Transformer-SSM knowledge distillation to a multimodal setting and propose the Distilled Audio-visual State-Space (DAVSS) model. The DAVSS model, 14M parameters, is 12 times smaller compared to transformer-based models such as CAV-MAE, and still outperforms them. DAVSS improves over the existing audio-visual models by: 1) Finer input resolution: using smaller patch sizes process the input, compensating for the smaller model size by increasing input sequence lengths. This is supported by the observation that a larger patch size results in lower performance. 2) Deeper joint modeling: utilizing a larger portion of the model (30%) for joint audio-visual processing, compared to <5% in CAV-MAE, enabling deeper cross-modal interaction without significantly increasing the computational cost associated with the concatenated audio-visual tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。