用类别嵌入提升时间序列预测精度，适配轻量与预训练模型。
QKCV Attention: Enhancing Time Series Forecasting with Static Categorical Embeddings for Both Lightweight and Pre-trained Foundation Models
- 在QKV注意力中加入静态类别嵌入C，强化类别信息捕捉。
- 在多个真实数据集上显著提升Transformer类模型的预测准确率。
- 仅更新类别嵌入即可高效微调预训练模型，节省计算资源。
在真实时间序列预测任务中，类别信息对捕捉数据内在模式至关重要。本文提出QKCV（Query-Key-Category-Value）注意力机制，作为传统QKV框架的扩展，引入静态类别嵌入C，以突出类别特异性信息。该模块为通用插件式设计，可提升多种基于注意力模型（如Vanilla Transformer、Informer、PatchTST、TFT）在多样真实数据集上的预测精度。此外，QKCV在微调单变量时间序列基础模型时表现出优异适应性，仅需更新静态嵌入C而保持预训练权重不变，显著降低计算开销，并实现更优的微调性能。
原文摘要 · Abstract (English)
In real-world time series forecasting tasks, category information plays a pivotal role in capturing inherent data patterns. This paper introduces QKCV (Query-Key-Category-Value) attention, an extension of the traditional QKV framework that incorporates a static categorical embedding C to emphasize category-specific information. As a versatile plug-in module, QKCV enhances the forecasting accuracy of attention-based models (e.g., Vanilla Transformer, Informer, PatchTST, TFT) across diverse real-world datasets. Furthermore, QKCV demonstrates remarkable adaptability in fine-tuning univariate time series foundation model by solely updating the static embedding C while preserving pretrained weights, thereby reducing computational overhead and achieving superior fine-tuning performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。