小模型也能精准识别情绪,轻量版模型效率更高
Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?

- 用知识蒸馏把大模型情绪理解能力迁移到小模型
- 在9个数据集上达到顶尖性能,推理速度提升数倍
- 适合机器人、手机等资源受限设备部署
近年来多模态大语言模型在多模态情绪识别(MER)任务中表现优异,可联合建模视频、音频与语言信息,实现可解释的描述生成。然而,这类模型通常参数量超过70亿,带来高昂计算成本并降低推理效率,阻碍其在机器人、移动设备等资源受限平台上的实时部署。本文质疑“更大模型必然更好”的假设,提出轻量级MER框架Light-MER,通过知识蒸馏将强大学习型教师模型的知识迁移至参数量低于10亿的学生模型,在保持丰富多模态情绪推理能力的同时显著提升部署效率。具体提出两项优化策略:(1) 结合切片沃尔什距离与隐藏状态对齐的最优传输损失;(2) 基于GRPO的多奖励优化策略,平衡性能与效率。在九个基准数据集上的大量实验表明,Light-MER在实现最先进性能的同时大幅提升推理效率,彰显小模型在未来的巨大潜力。代码已开源。
原文摘要 · Abstract (English)
Recent advances in multimodal large language models (MLLMs) have significantly improved the performance of multimodal emotion recognition (MER) and enabled interpretable description generation by jointly modeling video, audio, and language, etc. However, these performance improvements are often accompanied by an increase in model parameter size (e.g, at least 7B), which simultaneously incurs high computational costs and reduces inference efficiency, thereby hindering real-time deployment on resource-constrained platforms such as robots and mobile devices. This raises a fundamental question: do we really need the multimodal MER model larger than 1B parameters for high-quality MER? In this paper, we challenge the assumption that larger models are inherently necessary and proposes a lightweight MER framework (called Light-MER), which achieves better and faster multimodal sentiment understanding and recognition through knowledge distillation. It can transfer knowledge from a strong, large-scale teacher model to a lightweight sub-billion-parameter student model, aiming to preserve rich multimodal emotion reasoning and recognition while substantially improving deployment efficiency. Specifically, we introduce two new optimization strategies to enhance knowledge transfer: (1) a new optimal transport loss that combines Sliced Wasserstein Distance with hidden-state alignment, and (2) a new multi-reward optimization strategy based on GRPO that balances MER performance and efficiency, aimed at further enhancing the learning capabilities of student models. Extensive experiments on nine benchmark datasets demonstrate that Light-MER achieves state-of-the-art performance while significantly improving inference efficiency. This highlights the strong potential of small multimodal emotion language models for future research. Code is available at https://github.com/GAIR-Lab/Light-MER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。