用扩散Transformer建模车载视觉未来，实现更真实可控的自动驾驶仿真。
Diffusion Transformer World-Action Model for AV Scene Prediction
- 采用冻结编码器+潜空间扩散Transformer预测8秒后场景
- 扩散模型比回归方法在感知质量上提升4.8倍（KID 0.078 vs 0.375）
- 适合自动驾驶规划与仿真，尤其关注动作可控制性
面向自动驾驶的行动条件世界模型可基于自身规划动作预测未来相机画面，实现无需真实部署的规划与仿真。但现有紧凑可训练模型对未来预测模糊，且主流失真指标（如余弦相似度、SSIM）会误导优化方向：奖励模糊的平均回归结果而非真实预测。本文提出一种紧凑的潜空间世界模型，给定当前前视摄像头潜变量及一系列自车动作序列，通过冻结解码器生成最多8秒后的$256 \times 256$帧图像，基于nuScenes数据集150个保留场景进行评估。首先对比六种冻结编码器，含时序上下文的V-JEPA2将转向均方误差降低40%。随后训练潜空间扩散Transformer（DiT），通过对照诊断确定其四要素：空间令牌、$x_0$目标、残差锚定、采样与目标不确定性的匹配。在稳定扩散VAE的编-预-解码流程中揭示核心矛盾：传统失真指标偏好模糊均值，掩盖了扩散模型更接近真实帧分布的事实。基于Inception的FID与KID显示清晰的感知-失真前沿：扩散模型达到KID 0.078，优于回归模型的0.375（提升4.8倍）。引入可部署的训练衍生校准机制，无需测试时真实标签即可实用化。模型真正具备动作可控性（转向驱动场景位移，斯皮尔曼ρ=0.81，回归为-0.18）。发现单次传递运动受限于共享当前锚点，设计出仅1.7M参数的“跳跃”模型，恢复全真运动幅度（1.02×真实值），而单次模型捕捉不足一半。
原文摘要 · Abstract (English)
Action-conditioned world models let an autonomous vehicle predict future camera scenes from its own planned controls, enabling planning and simulation without real-world rollouts, but at compact, trainable scale the futures are ambiguous and the field's standard distortion metrics actively mislead: they reward a blurry regression mean over a realistic prediction. We confront this with a compact latent world model that, given the present front-camera latent and a sequence of ego-actions, predicts future scene latents a frozen decoder renders to $256 \times 256$ frames up to 8 seconds ahead, evaluated on 150 held-out nuScenes scenes. We first benchmark where to predict: across six frozen encoders spanning four representation families, V-JEPA2 with temporal context reduces steering RMSE by 40% over the best single-frame encoder. We then train a latent Diffusion Transformer (DiT) and, through a controlled diagnosis, identify the four ingredients it needs: spatial tokens, the $x_0$ objective, residual anchoring, and sampling matched to target uncertainty. In a Stable-Diffusion-VAE encode-predict-decode pipeline we expose the central tension: distortion metrics (cosine similarity, SSIM) favor the blurry mean, masking that the diffusion model is far closer to the real frame distribution. Inception-based FID and KID reveal a clean perception-distortion frontier: diffusion attains KID 0.078 versus 0.375 for regression ($4.8\times$ better), and a deployable train-derived calibration makes this practical without test-time ground truth. The model is genuinely action-controllable (steering drives scene displacement, Spearman $ρ= 0.81$, vs $-0.18$ for regression). We trace limited single-pass motion to a shared-present anchor and engineer a compact 1.7M-parameter "jump" model that recovers full ground-truth motion magnitude ($1.02\times$ GT), where single-pass models capture less than half.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。