arXiv:2506.10630cs.LGcs.AI2025-06被引 35

用强化学习训练大模型慢思考,提升时间序列预测准确性

Time Series Forecasting via Reasoning: A Slow-Thinking Approach with Reinforcement Fine-Tuned LLMs

  • 分两阶段微调大模型,先监督学习再强化学习
  • 设计细粒度多目标奖励机制,提升推理路径质量
  • 适合需要深度时序推理的科研与工业场景

为推进时间序列预测(TSF),现有方法从统计模型发展到数据驱动的深度学习架构。尽管有效,多数仍依赖快速思维范式——仅提取历史模式并映射未来值,缺乏显式的中间推理过程。新兴的慢思考大模型(如OpenAI-o1)展现出强大的多步推理能力,提供新思路。但仅靠提示工程存在计算成本高、隐私风险及领域推理深度不足等局限。为此,我们提出Time-R1,一种两阶段强化微调框架,旨在增强大模型在时间序列预测中的多步推理能力。第一阶段进行监督微调以实现初步适配,第二阶段采用强化学习提升模型泛化性。特别地,我们设计了针对时序预测的细粒度多目标奖励,并引入GRIP(基于组的相对重要性策略优化),通过非均匀采样促进模型对有效推理路径的探索与优化。实验表明,Time-R1在多个数据集上显著提升预测性能。

原文摘要 · Abstract (English)

To advance time series forecasting (TSF), various methods have been proposed to improve prediction accuracy, evolving from statistical techniques to data-driven deep learning architectures. Despite their effectiveness, most existing methods still adhere to a fast thinking paradigm-relying on extracting historical patterns and mapping them to future values as their core modeling philosophy, lacking an explicit thinking process that incorporates intermediate time series reasoning. Meanwhile, emerging slow-thinking LLMs (e.g., OpenAI-o1) have shown remarkable multi-step reasoning capabilities, offering an alternative way to overcome these issues. However, prompt engineering alone presents several limitations - including high computational cost, privacy risks, and limited capacity for in-depth domain-specific time series reasoning. To address these limitations, a more promising approach is to train LLMs to develop slow thinking capabilities and acquire strong time series reasoning skills. For this purpose, we propose Time-R1, a two-stage reinforcement fine-tuning framework designed to enhance multi-step reasoning ability of LLMs for time series forecasting. Specifically, the first stage conducts supervised fine-tuning for warmup adaptation, while the second stage employs reinforcement learning to improve the model's generalization ability. Particularly, we design a fine-grained multi-objective reward specifically for time series forecasting, and then introduce GRIP (group-based relative importance for policy optimization), which leverages non-uniform sampling to further encourage and optimize the model's exploration of effective reasoning paths. Experiments demonstrate that Time-R1 significantly improves forecast performance across diverse datasets.

时间序列大模型强化学习推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。