arXiv:2507.15470cs.LG2025-07中稿 · publication at IEE…被引 7

通过联邦学习融合视觉与生理信号,实现车内实时情绪识别。

FedMultiEmo: Real-Time Emotion Recognition via Multimodal Federated Learning

  • 在边缘设备上融合面部图像与心率等生理信号进行决策
  • 联合模型准确率达87%,媲美中心化训练且数据不离本地
  • 适合关注隐私与实时性的车载智能系统开发者

车内情绪识别是自适应驾驶辅助系统和乘员安全的基础。但实际部署面临三大挑战:(1)模态脆弱性——光照不足与遮挡会降低视觉方法效果;(2)生理差异性——心率与皮肤电导模式个体差异大;(3)隐私风险——集中式训练需传输敏感数据。为此,我们提出 FedMultiEmo,一种隐私保护的多模态联邦学习框架,采用决策级融合:使用卷积神经网络从人脸图像提取视觉特征,用随机森林分类心率、皮肤电导与皮肤温度等生理信号。该框架包含三个核心部分:(1)基于多数投票的多模态联邦学习流程;(2)基于树莓派客户端与Flower服务器的端到端原型系统;(3)按本地数据量加权的个性化联邦平均方案。在FER2013和自建生理数据集上评估,联邦CNN达77%准确率,随机森林74%,融合后达87%,与中心化基线相当,且所有原始数据均保留在本地。系统18轮收敛,单轮平均耗时120秒,每客户端内存占用低于200 MB。结果表明,FedMultiEmo为汽车场景下的实时、隐私友好的情绪识别提供了可行方案。

原文摘要 · Abstract (English)

In-vehicle emotion recognition underpins adaptive driver-assistance systems and, ultimately, occupant safety. However, practical deployment is hindered by (i) modality fragility - poor lighting and occlusions degrade vision-based methods; (ii) physiological variability - heart-rate and skin-conductance patterns differ across individuals; and (iii) privacy risk - centralized training requires transmission of sensitive data. To address these challenges, we present FedMultiEmo, a privacy-preserving framework that fuses two complementary modalities at the decision level: visual features extracted by a Convolutional Neural Network from facial images, and physiological cues (heart rate, electrodermal activity, and skin temperature) classified by a Random Forest. FedMultiEmo builds on three key elements: (1) a multimodal federated learning pipeline with majority-vote fusion, (2) an end-to-end edge-to-cloud prototype on Raspberry Pi clients and a Flower server, and (3) a personalized Federated Averaging scheme that weights client updates by local data volume. Evaluated on FER2013 and a custom physiological dataset, the federated Convolutional Neural Network attains 77% accuracy, the Random Forest 74%, and their fusion 87%, matching a centralized baseline while keeping all raw data local. The developed system converges in 18 rounds, with an average round time of 120 seconds and a per-client memory footprint below 200 MB. These results indicate that FedMultiEmo offers a practical approach to real-time, privacy-aware emotion recognition in automotive settings.

情绪识别联邦学习多模态车载系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。