arXiv:2509.21760cs.CV2025-09被引 6

用视频生成模型统一图像与视频任务,无需额外训练。

UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models

  • 将视觉任务转为视觉句子,用同一模型处理不同任务。
  • 仅用自然视频训练,跨模态和跨源任务表现良好。
  • 切换理解与生成任务只需反转输入顺序,极简高效。

大语言模型通过大规模语料训练,成功在单一生成框架内统一多种语言任务。受此启发,大型视觉模型(LVM)将该范式扩展至视觉领域,将任务组织为连续的视觉句子,以视觉提示作为上下文引导输出。然而,这种建模需在多模态和多源数据上进行任务特异性预训练,成本高且难以扩展至未见任务。鉴于预训练视频生成模型天然捕捉时间序列依赖性,我们探索更统一、可扩展的替代方案:预训练视频生成模型能否适应多样视觉任务?为此,我们提出UniVid框架,微调一个视频扩散变压器,在不进行任务特异性修改的情况下处理多种视觉任务。任务以视觉句子形式表示,上下文序列定义任务及期望输出模态。我们从两个角度评估UniVid的泛化能力:(1) 使用图像与视频组合的上下文进行跨模态推理,超越LVM的单模态设定;(2) 从自然数据到标注数据的跨源任务,无需多源预训练。尽管仅在自然视频数据上训练,UniVid在两种设置中均表现良好。值得注意的是,理解与生成任务可通过简单反转视觉句子顺序实现切换。这些发现表明,预训练视频生成模型有望成为视觉建模的可扩展统一基础。代码将发布于 https://github.com/CUC-MIPG/UniVid。

原文摘要 · Abstract (English)

Large language models, trained on extensive corpora, successfully unify diverse linguistic tasks within a single generative framework. Inspired by this, recent works like Large Vision Model (LVM) extend this paradigm to vision by organizing tasks into sequential visual sentences, where visual prompts serve as the context to guide outputs. However, such modeling requires task-specific pre-training across modalities and sources, which is costly and limits scalability to unseen tasks. Given that pre-trained video generation models inherently capture temporal sequence dependencies, we explore a more unified and scalable alternative: can a pre-trained video generation model adapt to diverse image and video tasks? To answer this, we propose UniVid, a framework that fine-tunes a video diffusion transformer to handle various vision tasks without task-specific modifications. Tasks are represented as visual sentences, where the context sequence defines both the task and the expected output modality. We evaluate the generalization of UniVid from two perspectives: (1) cross-modal inference with contexts composed of both images and videos, extending beyond LVM's uni-modal setting; (2) cross-source tasks from natural to annotated data, without multi-source pre-training. Despite being trained solely on natural video data, UniVid generalizes well in both settings. Notably, understanding and generation tasks can easily switch by simply reversing the visual sentence order in this paradigm. These findings highlight the potential of pre-trained video generation models to serve as a scalable and unified foundation for vision modeling. Our code will be released at https://github.com/CUC-MIPG/UniVid.

视频生成统一建模零样本迁移扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。