arXiv:2601.14259cs.CVcs.AI2026-01被引 1

融合视觉语音文本的云端情感识别模型,实现低延迟实时交互。

A Cloud-Based Cross-Modal Transformer for Emotion Recognition and Adaptive Human-Computer Interaction

  • 用跨模态注意力融合视觉、语音、文本特征,提升情感识别鲁棒性。
  • 在IEMOCAP等数据集上F1提升3.0%,交叉熵损失降低12.9%。
  • 云部署响应延迟仅128毫秒,适合智能客服、虚拟教学等场景。

情感识别是下一代人机交互的核心,使机器能够感知、理解并回应用户的情绪状态。现有系统多依赖单一模态(如面部表情、语音语调或文本情绪),在真实环境中泛化能力差。为此,本文提出基于云的跨模态变压器(CMT)框架,集成视觉(Vision Transformer)、音频(Wav2Vec2)和文本(BERT)预训练编码器,通过跨模态注意力捕捉异构特征间的复杂关联。借助Kubernetes与TensorFlow Serving的分布式云架构,系统实现可扩展、低延迟的情感识别。在IEMOCAP、MELD和AffectNet等基准数据集上的实验表明,该模型相比强基线提升F1-score 3.0%,交叉熵损失降低12.9%。云部署评估显示平均响应延迟为128毫秒,较传统基于Transformer的融合系统降低35%。结果验证了该框架在智能客服、虚拟辅导系统及情感计算界面中实现高效实时反馈的能力,标志着向云原生情感计算迈出了重要一步。

原文摘要 · Abstract (English)

Emotion recognition is a fundamental component of next-generation human-computer interaction (HCI), enabling machines to perceive, understand, and respond to users' affective states. However, existing systems often rely on single-modality analysis such as facial expressions, speech tone, or textual sentiment, resulting in limited robustness and poor generalization in real-world environments. To address these challenges, this study proposes a Cloud-Based Cross-Modal Transformer (CMT) framework for multimodal emotion recognition and adaptive human-computer interaction. The proposed model integrates visual, auditory, and textual signals using pretrained encoders (Vision Transformer, Wav2Vec2, and BERT) and employs a cross-modal attention mechanism to capture complex interdependencies among heterogeneous features. By leveraging cloud computing infrastructure with distributed training on Kubernetes and TensorFlow Serving, the system enables scalable, low-latency emotion recognition for large-scale user interactions. Experiments conducted on benchmark datasets including IEMOCAP, MELD, and AffectNet demonstrate that the CMT achieves state-of-the-art performance, improving the F1-score by 3.0 percent and reducing cross-entropy loss by 12.9 percent compared to strong multimodal baselines. Additionally, cloud deployment evaluations show an average response latency of 128 ms, representing a 35 percent reduction compared with conventional transformer-based fusion systems. These results confirm that the proposed framework enables efficient, real-time emotion recognition and adaptive feedback in applications such as intelligent customer service, virtual tutoring systems, and affective computing interfaces, marking an important step toward cloud-native affective computing and emotionally intelligent interactive systems.

情感识别跨模态云端推理人机交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。