arXiv:2503.20748cs.CV2025-03CVPR被引 5

统一时空建模框架,一模型搞定十类任务,训练成本更低。

UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines

  • 用视觉与图文预训练构建通用基础,再联合微调提升任务适应性。
  • 在跨4学科10任务数据集上,单模型支持全部任务,降低多域训练开销。
  • 创新使用连续插值的专家混合机制,增强跨域学习能力,适合多任务研究者。

传统时空模型依赖任务特定架构,受限于领域设计需求,泛化性和可扩展性差。本文提出基于Transformer的统一时空建模框架UniSTD,受近期基础模型启发,采用两阶段预训练-适配范式。研究表明,在2D视觉和视觉-文本数据集上进行无任务预训练,可建立可泛化的时空学习基础,随后在时空数据集上进行联合微调以增强任务特异性。为提升跨域学习能力,框架引入基于分数插值的秩自适应专家混合机制,使离散变量可在连续空间优化;同时设计显式时间模块以捕捉时序动态。在覆盖4个学科10个任务的大规模数据集上评估,证明统一时空模型可实现可扩展的跨任务学习,支持单模型同时处理10项任务,并显著降低多领域应用的训练成本。代码将开源。

原文摘要 · Abstract (English)

Traditional spatiotemporal models generally rely on task-specific architectures, which limit their generalizability and scalability across diverse tasks due to domain-specific design requirements. In this paper, we introduce \textbf{UniSTD}, a unified Transformer-based framework for spatiotemporal modeling, which is inspired by advances in recent foundation models with the two-stage pretraining-then-adaption paradigm. Specifically, our work demonstrates that task-agnostic pretraining on 2D vision and vision-text datasets can build a generalizable model foundation for spatiotemporal learning, followed by specialized joint training on spatiotemporal datasets to enhance task-specific adaptability. To improve the learning capabilities across domains, our framework employs a rank-adaptive mixture-of-expert adaptation by using fractional interpolation to relax the discrete variables so that can be optimized in the continuous space. Additionally, we introduce a temporal module to incorporate temporal dynamics explicitly. We evaluate our approach on a large-scale dataset covering 10 tasks across 4 disciplines, demonstrating that a unified spatiotemporal model can achieve scalable, cross-task learning and support up to 10 tasks simultaneously within one model while reducing training costs in multi-domain applications. Code will be available at https://github.com/1hunters/UniSTD.

时空建模统一框架多任务学习Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。