arXiv:2512.01342cs.CV2025-12被引 6

无需图文监督,通过新架构实现视频通用表征学习

InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervision

  • 拆解编码器-解码器为三阶段框架,引入潜空间世界模型
  • 两阶段预训练提升语义一致性与细节保留,性能超越现有方法
  • 适合追求无监督视频表征的科研与工业应用

大规模视频-文本预训练虽表现优异,但依赖噪声大、语义覆盖有限的合成描述,常忽略物体运动、三维几何和物理线索等隐含世界知识。相比之下,掩码视频建模(MVM)直接利用时空结构,但在通用任务上仍落后于文本监督方法。我们发现差距源于被忽视的架构问题:像素级重建收敛困难且低层需求与高层语义冲突,而潜变量预测易导致捷径学习。为此,我们提出将传统编码器-解码器拆分为编码器-预测器-解码器(EPD)框架,其中预测器作为潜空间世界模型,并设计两阶段预训练方案,在保持细节的同时构建语义一致的潜空间。第一阶段采用条件扩散解码器并注入可靠的图像级语义先验,增强语义表达与收敛性;第二阶段在冻结第一阶段目标的基础上预测潜变量,缓解捷径学习。模型仅使用公开无标签视频训练,在多个基准测试中达到当前最优,为通用视频表示学习提供可扩展路径。

原文摘要 · Abstract (English)

Large-scale video-text pretraining achieves strong performance but depends on noisy, synthetic captions with limited semantic coverage, often overlooking implicit world knowledge such as object motion, 3D geometry, and physical cues. In contrast, masked video modeling (MVM) directly exploits spatiotemporal structures but trails text-supervised methods on general tasks. We find this gap arises from overlooked architectural issues: pixel-level reconstruction struggles with convergence and its low-level requirement often conflicts with semantics, while latent prediction often encourages shortcut learning. To address these, we disentangle the traditional encoder-decoder design into an Encoder-Predictor-Decoder (EPD) framework, where the predictor acts as a latent world model, and propose InternVideo-Next, a two-stage pretraining scheme that builds a semantically consistent yet detail-preserving latent space for this world model. First, conventional linear decoder in pixel MVM enforces the predictor output latent to be linearly projected to, thus separable in pixel space, causing the conflict with semantic abstraction. Our Stage 1 proposes a conditional diffusion decoder and injects reliable image-level semantic priors to enhance semantics and convergence, thus bridging pixel-level fidelity with high-level semantic abstraction. Stage 2 further learns world knowledge by predicting frozen Stage 1 targets within this space, mitigating shortcut learning. Trained on public, unlabeled videos, InternVideo-Next achieves state-of-the-art results across benchmarks and provides a scalable path toward general video representation learning.

视频表征无监督学习世界模型自监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。