arXiv:2411.10087cs.LGcs.AI2024-11中稿 · publication in IEE…被引 5

提出PFML方法,解决时间序列自监督学习中的表征坍塌问题。

PFML: Self-Supervised Learning of Time-Series Data Without Representation Collapse

  • 通过预测掩码嵌入对应的统计函数值来避免表征坍塌
  • 在3种真实数据上分类准确率优于对比方法
  • 适合临床传感器等新模态时间序列数据应用

自监督学习(SSL)利用数据内在结构生成监督信号,但常出现表征坍塌问题,即模型输出恒定的输入无关特征。本文提出针对时间序列数据的新算法PFML,不直接预测掩码信号或其嵌入,而是基于未掩码嵌入序列预测对应掩码嵌入的统计函数值。该方法有效避免表征坍塌,可直接应用于不同时间序列领域,如临床传感器数据。我们在三种真实场景中验证:多传感器惯性数据的婴儿姿势动作分类、语音情绪识别和脑电睡眠分期。结果表明,PFML优于概念相似的基线方法和对比学习方法,性能与当前最优方法相当,且更简单、无表征坍塌问题。

原文摘要 · Abstract (English)

Self-supervised learning (SSL) is a data-driven learning approach that utilizes the innate structure of the data to guide the learning process. In contrast to supervised learning, which depends on external labels, SSL utilizes the inherent characteristics of the data to produce its own supervisory signal. However, one frequent issue with SSL methods is representation collapse, where the model outputs a constant input-invariant feature representation. This issue hinders the potential application of SSL methods to new data modalities, as trying to avoid representation collapse wastes researchers' time and effort. This paper introduces a novel SSL algorithm for time-series data called Prediction of Functionals from Masked Latents (PFML). Instead of predicting masked input signals or their latent representations directly, PFML operates by predicting statistical functionals of the input signal corresponding to masked embeddings, given a sequence of unmasked embeddings. The algorithm is designed to avoid representation collapse, rendering it straightforwardly applicable to different time-series data domains, such as novel sensor modalities in clinical data. We demonstrate the effectiveness of PFML through complex, real-life classification tasks across three different data modalities: infant posture and movement classification from multi-sensor inertial measurement unit data, emotion recognition from speech data, and sleep stage classification from EEG data. The results show that PFML is superior to a conceptually similar SSL method and a contrastive learning-based SSL method. Additionally, PFML is on par with the current state-of-the-art SSL method, while also being conceptually simpler and without suffering from representation collapse.

时间序列自监督学习表征坍塌生理信号

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。