arXiv:2606.05769cs.CV2026-06被引 3

让模型在推理时交替使用视觉隐向量和文本,提升视频预测准确性。

Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction

论文配图:Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction
图 1 · 摘自论文原文
  • 采用语言与视觉隐向量交替的推理框架,避免信息丢失。
  • 在FutureBench上将准确率从61.0提升至85.4,超过前人最佳10.4分。
  • 适合需要高精度视频未来状态预测的研究者或应用开发者。

视频事件预测(VEP)要求模型从部分视频证据中推断未观测的未来状态。现有视频多模态大模型通常将中间推理过程以文本形式表达:一旦视觉证据被转为文本,精细的运动、几何和交互线索可能丢失,导致看似合理但缺乏视觉依据的幻觉。本文提出Future-L1,一种交错式潜在视觉推理框架,使多模态大模型在自回归解码中交替使用语言标记与连续潜在视觉段。为训练该能力,构建了包含50,000个样本的Future-L1-50K数据集,筛选出未来视觉提示有助于预测的案例,并对齐潜在状态与未来帧嵌入;进一步通过LA-DAPO(一种带有结果对比和时序多样性奖励的潜空间感知强化学习目标)优化采样潜轨迹。Future-L1在两个基准测试上均取得新最优性能:在FutureBench上,将Qwen3-VL-8B的得分从61.0提升至85.4,超越先前最佳方法Video-CoE达10.4分;在TwiFF-Bench上,平均得分从2.44升至3.04。结果表明,将中间视觉语义保留在潜空间而非全转为文本,更有利于面向未来的视频推理。

原文摘要 · Abstract (English)

Video event prediction (VEP) requires models to infer unobserved future states from partial video evidence. Existing video MLLMs usually verbalize intermediate future reasoning in text space: once visual evidence is verbalized, fine-grained motion, geometry, and interaction cues can be lost, leading to plausible but visually ungrounded hallucinations. We introduce Future-L1, an interleaved latent visual reasoning framework that lets an MLLM alternate between language tokens and continuous latent visual spans during autoregressive decoding. To train this capability, we construct Future-L1-50K by selecting examples where future visual hints help prediction and align latent states to future-frame embeddings, then further optimize sampled latent trajectories with LA-DAPO, a latent-aware RL objective with outcome-contrastive and temporal-diversity rewards. Future-L1 achieves new state-of-the-art results on both benchmarks: on FutureBench, it improves Qwen3-VL-8B from 61.0 to 85.4 and exceeds the previous best Video-CoE by 10.4 points; on TwiFF-Bench, it improves the average score from 2.44 to 3.04. These results suggest that future-oriented video reasoning benefits from preserving intermediate visual semantics in latent space rather than translating every reasoning step into text.

视频预测视觉推理多模态潜空间

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。