arXiv:2410.07299cs.LGcs.AI2024-10被引 7

小模型实现顶级时序特征,适配穿戴设备等低资源场景。

OTIS: Learning High-Quality Time Series Features With Tiny Encoders

  • 针对时序特性优化预训练,用710万参数达大模型性能。
  • 在162个任务上超越54倍大的模型,内存、能耗、延迟大幅降低。
  • 适合部署在可穿戴设备、工业传感器等资源受限系统。

我们提出OTIS,一个开源的时序编码器,可在任意系统上生成高质量时序特征,包括资源受限的可穿戴设备和工业传感器。当前高性能通用编码器依赖规模定律,使用大型编码器记忆多领域训练数据的异构分布,但这种对规模的依赖限制了实际应用,因严格内存、功耗和延迟约束使部署不可行。令人惊讶的是,我们将标准掩码建模预训练适配时序特性后,仅用710万参数的微型编码器,在162项任务中达到54倍大模型的顶尖性能,同时内存减少10倍、能耗降低43倍、延迟下降37倍。为避免容量代价,我们引入三个新组件:(1) 域感知分词器,解决多域数据中的语义冲突;(2) 双重掩码策略,捕捉时空结构与时间因果性;(3) 结构感知目标,分离特征学习与噪声建模。结果表明,OTIS生成的高质量特征不仅支持判别任务达到顶尖表现,还能以极低成本无缝扩展至生成任务。为推动高能时序特征在各类系统上的普及,我们公开代码与预训练权重。

原文摘要 · Abstract (English)

We introduce OTIS, an open time series encoder that yields high-quality time series features for downstream deployment on any system, including resource-constrained wearables and industrial sensors. Currently, the development of powerful general-purpose encoders relies on the scaling laws hypothesis, using large encoder sizes to memorise the heterogeneous distributions of multi-domain training data. However, this reliance on scale creates a barrier to real-world utility, rendering deployment on resource-constrained systems infeasible due to strict memory, energy, and latency constraints. Surprisingly, we find that tailoring standard masked modelling pre-training to time series properties yields a tiny $7.1\,$M encoder that matches the state-of-the-art performance of $54\times$ larger encoders across $162$ tasks, while requiring $10\times$ less memory, $43\times$ less energy, and $37\times$ lower latency. To achieve this without the capacity tax, we introduce three novel components: (1) a domain-aware tokeniser to resolve conflicting semantics within multi-domain training data; (2) a dual masking strategy to capture spatiotemporal structures and temporal causality; and (3) a structure-aware objective to decouple feature learning from modelling noise. Consequently, OTIS produces high-quality time series features that enable state-of-the art performance in discriminative tasks and even extend seamlessly to generative tasks at minimal additional cost. To democratise access to powerful time series features on any system, we release our code and pre-trained weights.

时序建模轻量化模型特征提取边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。