轻量高效多模态情感分析新框架,融合更深入、参数更少。
PGF-Net: A Progressive Gated-Fusion Framework for Efficient Multimodal Sentiment Analysis
- 分层渐进融合:文本动态查询音视频特征,实现深层上下文感知融合
- 仅309万参数下达成MAE 0.691、F1 86.9%的领先性能
- 适合资源受限场景,兼具高精度与可解释性,适合多模态应用开发
我们提出PGF-Net(Progressive Gated-Fusion Network),一种高效且可解释的多模态情感分析深度学习框架。其核心创新包括:首先,提出渐进式层内融合范式,通过交叉注意力机制使文本表示在Transformer编码器深层动态查询并融合音频与视觉特征,实现更深层次的上下文依赖融合;其次,引入自适应门控仲裁机制,动态平衡原始语言信息与新融合的多模态上下文,确保融合稳定且抗噪声;最后,采用混合参数高效微调策略,结合LoRA全局适配与后融合适配器局部优化,显著减少可训练参数。该框架集成于分层编码结构中,实现深度、动态、可解释的多模态情感分析。在MOSI数据集上的实验表明,PGF-Net达到当前最优性能:平均绝对误差(MAE)为0.691,F1分数为86.9%,且仅需309万可训练参数,展现出性能与计算效率的卓越平衡。
原文摘要 · Abstract (English)
We introduce PGF-Net (Progressive Gated-Fusion Network), a novel deep learning framework designed for efficient and interpretable multimodal sentiment analysis. Our framework incorporates three primary innovations. Firstly, we propose a Progressive Intra-Layer Fusion paradigm, where a Cross-Attention mechanism empowers the textual representation to dynamically query and integrate non-linguistic features from audio and visual streams within the deep layers of a Transformer encoder. This enables a deeper, context-dependent fusion process. Secondly, the model incorporates an Adaptive Gated Arbitration mechanism, which acts as a dynamic controller to balance the original linguistic information against the newly fused multimodal context, ensuring stable and meaningful integration while preventing noise from overwhelming the signal. Lastly, a hybrid Parameter-Efficient Fine-Tuning (PEFT) strategy is employed, synergistically combining global adaptation via LoRA with local refinement through Post-Fusion Adapters. This significantly reduces trainable parameters, making the model lightweight and suitable for resource-limited scenarios. These innovations are integrated into a hierarchical encoder architecture, enabling PGF-Net to perform deep, dynamic, and interpretable multimodal sentiment analysis while maintaining exceptional parameter efficiency. Experimental results on MOSI dataset demonstrate that our proposed PGF-Net achieves state-of-the-art performance, with a Mean Absolute Error (MAE) of 0.691 and an F1-Score of 86.9%. Notably, our model achieves these results with only 3.09M trainable parameters, showcasing a superior balance between performance and computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。