用向量注入提升大模型时间序列预测能力,不需训练且计算开销小。
Enhancing Large Language Models for Time-Series Forecasting via Vector-Injected In-Context Learning
- 通过可学习的上下文向量适配器从样本中提取压缩信息
- 将向量注入模型每层,提升预测性能且不增加提示长度
- 适合追求高效高精度时间序列预测的研究者与工程师
全球网络需要可靠的预测能力以应对用户行为和使用模式的变化。时间序列预测(TSF)是实现这一目标的关键手段。近年来,基于大语言模型的时间序列预测方法(LLM4TSF)取得了良好表现。然而,预训练语料与时间序列数据之间存在显著差异,直接应用大模型难以保证预测质量;微调虽能缓解此问题,但常带来巨大计算开销。因此,LLM4TSF面临预测性能与计算开销的双重挑战。为此,我们探索在冻结所有大模型参数的前提下,提升其时间序列预测性能的方法。受上下文学习(ICL)启发,提出向量注入式上下文学习(LVICL)。LVICL利用向量注入的ICL机制,将示例信息注入冻结的大模型中,激发其上下文学习能力,从而提升在相关任务(即时间序列预测)上的表现。具体地,先使用大模型与可学习的上下文向量适配器,自适应地从多个示例中提取上下文向量,该向量包含压缩的、与示例相关的特征信息。随后,在前向传播过程中,将该向量注入大模型每一层,以增强预测性能。相较于传统ICL将示例加入提示的方式,我们的方法不增加提示长度;同时,自适应地从示例中提取上下文向量,可抑制对预测有害的成分,从而提升模型性能。大量实验验证了该方法的有效性。
原文摘要 · Abstract (English)
The World Wide Web needs reliable predictive capabilities to respond to changes in user behavior and usage patterns. Time series forecasting (TSF) is a key means to achieve this goal. In recent years, the large language models (LLMs) for TSF (LLM4TSF) have achieved good performance. However, there is a significant difference between pretraining corpora and time series data, making it hard to guarantee forecasting quality when directly applying LLMs to TSF; fine-tuning LLMs can mitigate this issue, but often incurs substantial computational overhead. Thus, LLM4TSF faces a dual challenge of prediction performance and compute overhead. To address this, we aim to explore a method for improving the forecasting performance of LLM4TSF while freezing all LLM parameters to reduce computational overhead. Inspired by in-context learning (ICL), we propose LVICL. LVICL uses our vector-injected ICL to inject example information into a frozen LLM, eliciting its in-context learning ability and thereby enhancing its performance on the example-related task (i.e., TSF). Specifically, we first use the LLM together with a learnable context vector adapter to extract a context vector from multiple examples adaptively. This vector contains compressed, example-related information. Subsequently, during the forward pass, we inject this vector into every layer of the LLM to improve forecasting performance. Compared with conventional ICL that adds examples into the prompt, our vector-injected ICL does not increase prompt length; moreover, adaptively deriving a context vector from examples suppresses components harmful to forecasting, thereby improving model performance. Extensive experiments demonstrate the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。