arXiv:2410.18072cs.CV2024-10被引 1.3k

提出首个面向世界模拟器的双维度评估框架,解决视频生成模型评价难题。

WorldSimBench: Towards Video Generation Models as World Simulators

  • 构建分层功能分类体系与双评估框架,涵盖感知与操作双重维度。
  • 引入HF-Embodied数据集,训练人类偏好评估器实现视觉保真度量化。
  • 在开放环境、自动驾驶和机器人操作中验证生成视频的动作一致性。

近期预测模型在物体与场景未来状态预测方面表现卓越,但缺乏基于内在特征的分类体系,制约了其发展。现有基准难以从具身视角有效评估高能力、高度具身的预测模型。本文提出世界模拟器功能分层体系,并构建首个双维度评估框架WorldSimBench,包含显式感知评估与隐式操控评估。显式感知评估引入基于细粒度人类反馈的HF-Embodied数据集,训练人类偏好评估器,以视觉视角评估世界模拟器的视觉保真度;隐式操控评估则通过动态环境中生成视频能否准确转化为正确控制信号,检验视频-动作一致性。评估覆盖开放具身环境、自动驾驶与机器人操作三类典型场景。结果为视频生成模型的进一步创新提供关键洞见,推动世界模拟器向具身人工智能迈进。

原文摘要 · Abstract (English)

Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence.

视频生成世界模拟器具身智能评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。