arXiv:2507.11558cs.CVcs.AI2025-07被引 1

用视觉模型做时空预测,通过双路结构+动态提示提升效果

Reprogramming Vision Foundation Models for Spatio-Temporal Forecasting

  • 将视觉大模型改造为双分支输入,融合原始数据与动态流信息
  • 在10个数据集上超越现有方法,支持多种视觉模型骨干
  • 无需修改主干网络,适合想快速迁移视觉模型的从业者

基础模型在自然语言处理和计算机视觉中取得显著成功,展现出建模复杂模式的强大能力。尽管已有研究尝试将大语言模型用于时间序列预测,但其主要捕捉一维序列依赖,难以建模时空预测所需的丰富时空相关性。本文提出一种新框架ST-VFM,系统性地重编程视觉基础模型(VFMs)以实现通用时空预测。虽然VFMs具备强大的空间先验,但在应用于时空任务时面临两大挑战:(1) 缺乏内在的时间建模能力;(2) 视觉数据与时空数据之间的模态差距。为此,ST-VFM采用双分支架构,整合原始时空输入与辅助的时空流输入,其中流编码轻量级的时间差信号,可解释为动态空间线索。为有效处理双分支输入,引入两个专门的重编程阶段:预VFM重编程阶段使用时序感知令牌适配器嵌入时间上下文,并将两分支对齐至VFM兼容特征空间;后VFM重编程阶段引入双边交叉提示协调模块,通过提示驱动条件实现分支间的动态交互,从而在不修改冻结的VFM主干的前提下增强联合表征学习。在十个时空数据集上的大量实验表明,ST-VFM优于当前最优基线,在不同VFM骨干(如DINO、CLIP、DEIT)下均表现出色且鲁棒,确立了其作为通用时空预测框架的强有效性。

原文摘要 · Abstract (English)

Foundation models have achieved remarkable success in natural language processing and computer vision, demonstrating strong capabilities in modeling complex patterns. While recent efforts have explored adapting large language models (LLMs) for time-series forecasting, LLMs primarily capture one-dimensional sequential dependencies and struggle to model the richer spatio-temporal (ST) correlations essential for accurate ST forecasting. In this paper, we present \textbf{ST-VFM}, a novel framework that systematically reprograms Vision Foundation Models (VFMs) for general-purpose spatio-temporal forecasting. While VFMs offer powerful spatial priors, two key challenges arise when applying them to ST tasks: (1) the lack of inherent temporal modeling capacity and (2) the modality gap between visual and ST data. To address these, ST-VFM adopts a \emph{dual-branch architecture} that integrates raw ST inputs with auxiliary ST flow inputs, where the flow encodes lightweight temporal difference signals interpretable as dynamic spatial cues. To effectively process these dual-branch inputs, ST-VFM introduces two dedicated reprogramming stages. The \emph{pre-VFM reprogramming} stage applies a Temporal-Aware Token Adapter to embed temporal context and align both branches into VFM-compatible feature spaces. The \emph{post-VFM reprogramming} stage introduces a Bilateral Cross-Prompt Coordination module, enabling dynamic interaction between branches through prompt-based conditioning, thus enriching joint representation learning without modifying the frozen VFM backbone. Extensive experiments on ten spatio-temporal datasets show that ST-VFM outperforms state-of-the-art baselines, demonstrating effectiveness and robustness across VFM backbones (e.g., DINO, CLIP, DEIT) and ablation studies, establishing it as a strong general framework for spatio-temporal forecasting.

时空预测视觉模型大模型重编程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。