构建视频生成模型的全方位评估体系,精准定位性能短板。
VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models
- 将视频质量拆解为16个独立维度,逐项量化评估
- 通过人类偏好数据验证评测结果与主观感受一致
- 支持文生视频、图生视频,适合研究者和开发者使用
视频生成技术快速发展,但评估仍面临挑战。现有指标与人类感知不一致,且难以指导未来模型改进。为此,我们提出VBench++,一个全面的视频生成评估基准套件,将视频生成质量分解为16个层次化、解耦的维度(如主体身份一致性、运动流畅性、时间闪烁、空间关系等),每个维度配备定制提示与评估方法。该基准具备五大优势:1)全面性:覆盖16个关键维度,细粒度指标可揭示模型优劣;2)人因对齐:提供人类偏好标注数据集,验证各维度评估与主观感知的一致性;3)洞察力:分析当前模型在不同内容类型下的表现差异及视频与图像生成模型间的差距;4)通用性:支持文本到视频与图像到视频生成,引入自适应长宽比高质量图像套件,实现跨场景公平评估;5)开源持续更新:全面开源,并动态添加新模型至排行榜,推动领域发展。
原文摘要 · Abstract (English)
Video generation has witnessed significant advancements, yet evaluating these models remains a challenge. A comprehensive evaluation benchmark for video generation is indispensable for two reasons: 1) Existing metrics do not fully align with human perceptions; 2) An ideal evaluation system should provide insights to inform future developments of video generation. To this end, we present VBench, a comprehensive benchmark suite that dissects "video generation quality" into specific, hierarchical, and disentangled dimensions, each with tailored prompts and evaluation methods. VBench has several appealing properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in video generation (e.g., subject identity inconsistency, motion smoothness, temporal flickering, and spatial relationship, etc). The evaluation metrics with fine-grained levels reveal individual models' strengths and weaknesses. 2) Human Alignment: We also provide a dataset of human preference annotations to validate our benchmarks' alignment with human perception, for each evaluation dimension respectively. 3) Valuable Insights: We look into current models' ability across various evaluation dimensions, and various content types. We also investigate the gaps between video and image generation models. 4) Versatile Benchmarking: VBench++ supports evaluating text-to-video and image-to-video. We introduce a high-quality Image Suite with an adaptive aspect ratio to enable fair evaluations across different image-to-video generation settings. Beyond assessing technical quality, VBench++ evaluates the trustworthiness of video generative models, providing a more holistic view of model performance. 5) Full Open-Sourcing: We fully open-source VBench++ and continually add new video generation models to our leaderboard to drive forward the field of video generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。