测试现成视频模型在远程操控预测显示中的表现，发现均无法兼顾精度与实时性。
Towards Generative Predictive Display for Vision-Based Teleoperation: A Zero-Shot Benchmark of Off-the-Shelf Video Models
- 用仿真数据构建零样本基准，评估五种生成式视频模型的预测能力
- 所有模型在高精度、低延迟和稳定误差方面均未达标，规模扩大反致性能下降
- 适合研究远程操控视觉延迟问题或生成模型实际部署挑战的学者
远程操控系统受通信延迟制约，影响情境感知与控制表现。预测显示通过呈现当前视觉状态估计值而非延迟观测来缓解此问题。尽管生成式视频模型在高质量视频合成上取得进展，但其在低延迟预测显示场景下的适用性尚不明确。本文提出一个无需任务微调的零样本基准，评估五种公开发布的生成式视频模型在短时预测显示中的表现。问题被建模为基于回溯的未来帧预测，采用CARLA模拟器生成的驾驶数据构建统一评测流程。模型在两种分辨率和两种条件设置（多帧与单帧）下进行评估，指标包括预测准确率（平均绝对差）、每轮推断延迟、峰值显存占用及预测时序误差演化。结果显示，无一模型能同时实现低回溯误差、非发散的逐步误差行为和源帧率实时推理。模型规模或分辨率提升带来的改进有限，甚至出现倒置效果。这表明通用生成视频合成与远程操控预测显示需求之间存在差距，实际部署需依赖显式的短期时序监督、领域内适应或激进推理优化，而非直接使用现成模型。代码、配置与可视化结果已发布于项目主页：https://bimilab.github.io/paper-GenPD
原文摘要 · Abstract (English)
Teleoperation systems are fundamentally limited by communication latency, which degrades situational awareness and control performance. Predictive display aims to mitigate this limitation by presenting an estimate of the current visual state rather than delayed observations. While recent advances in generative video models enable high-quality video synthesis, their suitability for latency-sensitive predictive display remains unclear. This paper presents a zero-shot benchmark of off-the-shelf generative video models for short-horizon predictive display, without task-specific fine-tuning. We formulate the problem as rollout-based future frame prediction and develop a unified benchmarking pipeline using simulated driving data from the CARLA simulator. Five publicly released video models spanning transformer-based and diffusion-based families are evaluated across two resolutions and two conditioning regimes (multi-frame and single-frame). Performance is assessed using prediction accuracy (mean absolute difference), per-rollout latency, peak GPU memory usage, and temporal error evolution across the prediction horizon. On this zero-shot benchmark, no tested model simultaneously achieves low rollout error, non-divergent per-step error behavior, and real-time inference at the source frame rate. Increasing model scale or resolution yields limited and, in some cases, inverted improvements. These findings highlight a gap between general-purpose generative video synthesis and the requirements of predictive display in teleoperation, suggesting that practical deployment will require either explicit short-horizon temporal supervision, in-domain adaptation, or aggressive inference optimization rather than direct application of off-the-shelf models. Code, configurations, and qualitative results are released on the project page: https://bimilab.github.io/paper-GenPD
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。