arXiv:2512.15250cs.LGcs.AI2025-12被引 2

用基础模型+简单融合,提升多模态生理信号分析效果

Leveraging Foundational Models and Simple Fusion for Multi-modal Physiological Signal Analysis

  • 设计双掩码策略预训练心电图编码器,捕捉电极间依赖关系
  • 跨模态融合仅用嵌入拼接,仍达接近顶尖的情绪识别性能
  • 适合医疗健康与情感计算领域,尤其数据少时表现突出

心电图(ECG)和脑电图(EEG)提供互补的人体健康与认知信息,但因多模态标注数据有限及模态差异大,多模态融合困难。本文针对大尺度自监督心电图预训练,改进CBraMod编码器并引入双掩码策略以捕捉单导联内与导联间依赖。为克服挑战,采用预训练的CBraMod编码器处理EEG,并对称预训练心电图编码器,使各模态获得丰富基础表征。随后通过简单的嵌入拼接实现融合,使分类头学习跨模态交互,在有限多模态监督下仍实现有效下游学习。在情绪识别任务上,该方法达到近顶尖性能,证明精心设计的生理信号编码器即使使用简单融合,也能显著提升下游表现。结果表明,基础模型方法能有效利用生理信号的整体特性,为医疗健康与情感计算提供可扩展、低标签依赖且通用的解决方案。

原文摘要 · Abstract (English)

Physiological signals such as electrocardiograms (ECG) and electroencephalograms (EEG) provide complementary insights into human health and cognition, yet multi-modal integration is challenging due to limited multi-modal labeled data, and modality-specific differences . In this work, we adapt the CBraMod encoder for large-scale self-supervised ECG pretraining, introducing a dual-masking strategy to capture intra- and inter-lead dependencies. To overcome the above challenges, we utilize a pre-trained CBraMod encoder for EEG and pre-train a symmetric ECG encoder, equipping each modality with a rich foundational representation. These representations are then fused via simple embedding concatenation, allowing the classification head to learn cross-modal interactions, together enabling effective downstream learning despite limited multi-modal supervision. Evaluated on emotion recognition, our approach achieves near state-of-the-art performance, demonstrating that carefully designed physiological encoders, even with straightforward fusion, substantially improve downstream performance. These results highlight the potential of foundation-model approaches to harness the holistic nature of physiological signals, enabling scalable, label-efficient, and generalizable solutions for healthcare and affective computing.

多模态生理信号基础模型情绪识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。