让传感器数据理解语言,跨设备通用零样本推理
Learning Transferable Sensor Models via Language-Informed Pretraining
- 用语言对齐+对比学习,让传感器数据懂语义
- 零样本分类准确率77.14%,问答任务达64.83%
- 支持变长输入和不同采样率,无需重训练
现代传感系统产生大量未标注的多变量时间序列数据。这种数据的丰富性使自监督学习(SSL)成为学习可迁移表征的自然选择。然而,现有方法多以重建或预测为目标,难以捕捉下游分类与推理所需的语义结构。尽管近期传感器-语言对齐方法通过描述生成和零样本迁移提升了语义泛化能力,但其受限于固定传感器配置(如预设通道集、信号长度或时间分辨率),限制了跨领域适用性。为此,我们提出SLIP(Sensor Language-Informed Pretraining),一个开源框架,用于学习跨多样传感器设置的语言对齐表征。SLIP结合对比对齐与传感器条件化描述生成,同时支持判别性理解与生成式推理。通过利用预训练的仅解码器语言模型进行交叉注意力,并引入灵活的分块嵌入器,SLIP可在推理时支持不同时间分辨率和变长输入,无需额外训练。在11个数据集上,SLIP展现出卓越的零样本迁移、信号描述生成与问答性能:线性探测平均准确率达77.14%,相较强基线提升5.93%;传感器问答任务准确率为64.83%。
原文摘要 · Abstract (English)
Modern sensing systems generate large volumes of unlabeled multivariate time-series data. This abundance of unlabeled data makes self-supervised learning (SSL) a natural approach for learning transferable representations. However, most existing approaches are optimized for reconstruction or forecasting objectives and often fail to capture the semantic structure required for downstream classification and reasoning tasks. While recent sensor-language alignment methods improve semantic generalization through captioning and zero-shot transfer, they are limited to fixed sensor configurations, such as predefined channel sets, signal lengths, or temporal resolutions, which hinders cross-domain applicability. To address these gaps, we introduce \textbf{SLIP} (\textbf{S}ensor \textbf{L}anguage-\textbf{I}nformed \textbf{P}retraining), an open-source framework for learning language-aligned representations that generalize across diverse sensor setups. SLIP integrates contrastive alignment with sensor-conditioned captioning, facilitating both discriminative understanding and generative reasoning. By repurposing a pretrained decoder-only language model via cross-attention and introducing an elegant, flexible patch-embedder, SLIP supports different temporal resolutions and variable-length input at inference time without additional retraining. Across 11 datasets, SLIP demonstrates superior performance in zero-shot transfer, signal captioning, and question answering. It achieves a 77.14% average linear-probing accuracy, a 5.93% relative improvement over strong baselines, and reaches 64.83% accuracy in sensor-based question answering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。