用数据特性优化注意力，提升多变量时间序列建模效果
Dataset-Driven Channel Masks in Transformers for Multivariate Time Series
- 基于数据特征构建通道掩码，动态调整注意力权重
- 在多个数据集上显著提升模型性能，尤其对弱相关通道有效
- 适合处理多变量时间序列的科研与工业场景
近年来，大型时间序列（TS）数据集推动了基础模型在时序领域的应用。然而，现有方法多聚焦于架构改进，忽视了数据集特性的价值。本文提出部分通道依赖（PCD）概念，通过引入通道掩码（CMs）融合数据集特异性信息，优化Transformer中通道依赖（CD）的建模。CMs由相似性矩阵和可学习的领域参数构成，以元素乘法方式嵌入注意力矩阵。实验验证了该方法在多种任务和模型架构上的有效性，显著提升了多变量时间序列建模性能。代码已开源：https://github.com/YonseiML/pcd。
原文摘要 · Abstract (English)
Recent advancements in foundation models have been successfully extended to the time series (TS) domain, facilitated by the emergence of large-scale TS datasets. However, previous efforts have primarily Capturing channel dependency (CD) is essential for modeling multivariate time series (TS), and attention-based methods have been widely employed for this purpose. Nonetheless, these methods primarily focus on modifying the architecture, often neglecting the importance of dataset-specific characteristics. In this work, we introduce the concept of partial channel dependence (PCD) to enhance CD modeling in Transformer-based models by leveraging dataset-specific information to refine the CD captured by the model. To achieve PCD, we propose channel masks (CMs), which are integrated into the attention matrices of Transformers via element-wise multiplication. CMs consist of two components: 1) a similarity matrix that captures relationships between the channels, and 2) dataset-specific and learnable domain parameters that refine the similarity matrix. We validate the effectiveness of PCD across diverse tasks and datasets with various backbones. Code is available at this repository: https://github.com/YonseiML/pcd.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。