用Transformer融合文本音频视觉，精准预测情感强度。
Multimodal Sentiment Analysis on CMU-MOSEI Dataset using Transformer-based Models
- 三模态早期融合，各模态用BERT编码后拼接
- 7分类准确率97.87%，F1达0.9682，MAE仅0.1060
- 适合研究多模态情感分析或模型融合的开发者
本项目基于CMU-MOSEI数据集，采用基于Transformer的模型进行多模态情感分析，通过早期融合策略整合文本、音频和视觉信息。每个模态使用BERT编码器提取嵌入表示，再进行拼接后分类。模型在测试集上取得97.87%的7分类准确率和0.9682的F1分数，表明早期融合能有效捕捉跨模态交互。训练中采用Adam优化（学习率1e-4）、dropout（0.3）与早停机制，确保模型泛化能力。结果表明，Transformer架构在建模多模态情感方面具有显著优势，且平均绝对误差（MAE）低至0.1060，说明情感强度预测精度高。未来工作可比较不同融合策略或提升模型可解释性。该方法通过有效结合语言、声学和视觉线索实现情感分析。
原文摘要 · Abstract (English)
This project performs multimodal sentiment analysis using the CMU-MOSEI dataset, using transformer-based models with early fusion to integrate text, audio, and visual modalities. We employ BERT-based encoders for each modality, extracting embeddings that are concatenated before classification. The model achieves strong performance, with 97.87% 7-class accuracy and a 0.9682 F1-score on the test set, demonstrating the effectiveness of early fusion in capturing cross-modal interactions. The training utilized Adam optimization (lr=1e-4), dropout (0.3), and early stopping to ensure generalization and robustness. Results highlight the superiority of transformer architectures in modeling multimodal sentiment, with a low MAE (0.1060) indicating precise sentiment intensity prediction. Future work may compare fusion strategies or enhance interpretability. This approach utilizes multimodal learning by effectively combining linguistic, acoustic, and visual cues for sentiment analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。