先预测场景语义再生成视频，提升动态环境下的预测质量。
Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction
- 分两阶段预测：先建模语义结构，再生成视觉细节。
- 在驾驶数据集上显著提升语义一致性与画面真实感。
- 适合需要高语义连贯性的视频生成任务，如自动驾驶仿真。
准确的未来视频预测需兼顾高视觉保真度与一致的场景语义，尤其在自动驾驶等复杂动态环境中。我们提出 Re2Pix，一种分层视频预测框架，将预测过程分为两个阶段：语义表示预测与表示引导的视觉合成。不直接预测未来RGB帧，而是先在冻结视觉基础模型的特征空间中预测未来场景结构，再以这些预测表示为条件，驱动潜在扩散模型生成逼真图像。该分解使模型能先关注场景动态，再聚焦外观生成。训练与推理间存在真实表示与预测表示的不匹配问题，为此我们引入嵌套丢弃和混合监督两种条件策略，增强对非完美自回归预测的鲁棒性。在多个挑战性驾驶基准测试中，所提语义优先设计显著提升了时间语义一致性、感知质量与训练效率，优于强基线扩散模型。代码已开源：https://github.com/Sta8is/Re2Pix。
原文摘要 · Abstract (English)
Accurate future video prediction requires both high visual fidelity and consistent scene semantics, particularly in complex dynamic environments such as autonomous driving. We present Re2Pix, a hierarchical video prediction framework that decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis. Instead of directly predicting future RGB frames, our approach first forecasts future scene structure in the feature space of a frozen vision foundation model, and then conditions a latent diffusion model on these predicted representations to render photorealistic frames. This decomposition enables the model to focus first on scene dynamics and then on appearance generation. A key challenge arises from the train-test mismatch between ground-truth representations available during training and predicted ones used at inference. To address this, we introduce two conditioning strategies, nested dropout and mixed supervision, that improve robustness to imperfect autoregressive predictions. Experiments on challenging driving benchmarks demonstrate that the proposed semantics-first design significantly improves temporal semantic consistency, perceptual quality, and training efficiency compared to strong diffusion baselines. We provide the implementation code at https://github.com/Sta8is/Re2Pix
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。