arXiv:2605.28229cs.CVcs.AI2026-05

让视频生成专家各司其职,提升图像转视频的准确率。

VidPrism: Heterogeneous Mixture of Experts for Image-to-Video Transfer

论文配图:VidPrism: Heterogeneous Mixture of Experts for Image-to-Video Transfer
图 1 · 摘自论文原文
  • 用不同专长的专家分工处理视频的空间与时间特征。
  • 在多个视频识别数据集上达到顶尖性能,专家分工明确。
  • 适合研究视频理解、多模态模型优化的开发者参考。

随着大规模视觉-语言模型(VLMs)预训练技术的快速发展,将VLMs应用于视频理解——即图像到视频的迁移学习已成为主流范式。为实现卓越性能,近期进展普遍采用混合专家(MoE)机制以增强VLM的时间建模能力。然而,传统MoE设计存在专家同质化问题,所有专家均表现为通用型模型,无法高效从统一的视频流中学习时空特征。为此,本文提出VidPrism,一种新型异构时序混合专家框架。VidPrism通过部署功能专一的专家,实现角色分工,涵盖从空间理解到时序建模的不同任务。为精准供给专家输入,我们引入内容感知的多速率采样模块,动态生成语义丰富至运动聚焦的多类表征流。同时,设计动态双向融合机制,促进路径间信息协同,构建全面的视频表示。在多个视频识别基准上的大量实验表明,VidPrism实现了最先进的性能,并有效促进专家分化。代码已开源:https://github.com/Lrrrr549/VidPrism.git。

原文摘要 · Abstract (English)

With the rapid development of pre-training technologies, adapting large-scale Vision-Language Models (VLMs) for video understanding \emph{\ie} image-to-video transfer learning has become a dominant paradigm. To achieve superior performance, it raises as an effective strategy among recent advances to employ Mixture-of-Experts (MoE) to enhance VLMs' temporal modeling capabilities. However, conventional MoE designs suffer from expert homogenization, where all experts act as identical generalists, inefficiently learning spatio-temporal features from undifferentiated video streams. To overcome this problem, we propose VidPrism, a novel heterogeneous temporal Mixture-of-Experts framework. VidPrism pioneers a division of labor by deploying functionally specialized experts, each assuming a role ranging from spatial understanding to temporal modeling. To feed these specialists appropriately, we introduce a content-aware, multi-rate sampling module that dynamically generates streams ranging from semantically rich to motion-focused representations, providing specialized inputs for experts. Furthermore, a dynamic, bidirectional fusion mechanism enables synergistic information exchange between these pathways, leading to a comprehensive video representation. Extensive experiments on various video recognition benchmarks demonstrate that VidPrism achieves state-of-the-art performance and effectively fosters expert specialization. Our source code is available at \href{https://github.com/Lrrrr549/VidPrism.git}{https://github.com/Lrrrr549/VidPrism.git}.

视频生成混合专家多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。