arXiv:2412.04806cs.LGcs.AI2024-12被引 2

用邻域语义对齐和时序调制,让大模型更好融合文本与时间序列数据。

NeST: Neighborhood-aware semantic alignment and temporal modulation for LLM based time series forecasting

  • 构建邻域感知的文本原型,匹配时间序列局部特征。
  • 长周期预测均方误差降低1.2%,零样本泛化误差降4.9%。
  • 适合需要融合文本信息的复杂时序预测任务,如能源预测。

将预训练于离散文本的大语言模型(LLM)用于连续时间序列预测面临挑战。虽可通过微调实现适应,但如何有效整合文本与时间序列信息于提示中至关重要。现有方法多采用简单拼接或参数量大的交叉注意力机制,且时间序列嵌入依赖分解技术,难以捕捉复杂动态。为此,本文提出基于邻域感知语义对齐与时序调制的框架(NeST),设计新型文本融合时间序列提示以微调LLM。首先生成邻域感知的文本原型,优化其表示预训练词嵌入的局部邻域;其次通过最近邻对比目标,将原型与时间序列表示对齐,并检索最相关的k个原型;最后引入文本原型条件化的时序调制,利用检索到的原型自适应地缩放和偏移时间序列特征。大量实验表明,NeST在八个基准上持续优于当前最优方法,长周期预测平均降低1.2%的均方误差;在零样本场景下,平均误差下降4.9%。此外,在九个真实分布式光伏功率预测数据集上,平均决定系数R²提升3.3%。结果证明了该方法在多样时间序列预测任务中的有效性与泛化能力。

原文摘要 · Abstract (English)

Adapting Large Language Models (LLMs) trained on discrete text data, to forecast continuous time series signals is challenging. While finetuning the LLMs enables such adaptation, effectively integrating both textual and time series information in the prompt is critical. Current LLM-based time series forecasting methods combine the two modalities through simple concatenation or parameter heavy cross-attention. Moreover, existing methods embed time series data using decomposition techniques that may inadequately capture complex temporal dynamics. To address these limitations, we propose neighborhood-aware semantic alignment and temporal modulation based framework (NEST) to formulate a new text-integrated time series prompt to finetune the LLM. First, we generate neighborhood-aware text prototypes that are optimized to represent local neighborhoods of pretrained word token embeddings of the LLM. Second, we align them with temporal representations of the time series input using a nearest-neighbor contrastive objective, after which the top-k most relevant text prototypes are retrieved. Third, we introduce text prototype conditioned temporal modulation that uses the retrieved text prototypes to adaptively scale and shift time series features. Extensive experiments demonstrate that NeST consistently outperforms state-of-the-art methods across eight benchmarks, achieving an average 1.2\% reduction in MSE for long-term forecasting. In addition, it demonstrates strong generalization, yielding an average 4.9\% reduction in MSE in zero-shot forecasting. Beyond benchmark datasets, NeST also delivers robust performance on a real-world distributed photovoltaic power forecasting task across nine datasets, improving the average R$^2$ score by 3.3\%. These findings demonstrate the effectiveness and generalizability of NeST for adapting LLMs to diverse time series forecasting tasks.

时间序列大模型文本融合预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。