arXiv:2411.15435cs.CV2024-11被引 5

用场景图评估和反馈,提升图像生成的准确性和一致性。

What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation

  • 构建百万级场景图数据集,支持复杂场景生成评测
  • 提出基于多模态大模型的SGScore,更精准衡量生成质量
  • 迭代反馈机制显著提升图像与场景图的一致性

尽管文本到图像生成已广泛研究,但基于场景图生成图像仍相对不足,主要因难以准确建模空间关系与物体交互。为此,我们提出Scene-Bench,一个全面的基准测试框架,用于评估和提升自然场景生成的事实一致性。Scene-Bench包含MegaSG——一个包含一百万张标注场景图的大型数据集,支持多样复杂场景的模型训练与公平比较。同时,我们提出SGScore,一种利用多模态大语言模型(LLM)链式推理能力的新评估指标,可同时评估对象存在性和关系准确性,比传统指标如FID和CLIPScore更具有效性。在此评估框架基础上,我们开发了场景图反馈流水线,通过识别并修正图像与场景图之间的差异,实现迭代优化。大量实验表明,Scene-Bench相比现有基准提供了更全面、有效的评估能力,尤其适用于复杂场景生成。此外,我们的反馈策略显著提升了图像生成模型的事实一致性,推动可控图像生成领域发展。

原文摘要 · Abstract (English)

While text-to-image generation has been extensively studied, generating images from scene graphs remains relatively underexplored, primarily due to challenges in accurately modeling spatial relationships and object interactions. To fill this gap, we introduce Scene-Bench, a comprehensive benchmark designed to evaluate and enhance the factual consistency in generating natural scenes. Scene-Bench comprises MegaSG, a large-scale dataset of one million images annotated with scene graphs, facilitating the training and fair comparison of models across diverse and complex scenes. Additionally, we propose SGScore, a novel evaluation metric that leverages chain-of-thought reasoning capabilities of multimodal large language models (LLMs) to assess both object presence and relationship accuracy, offering a more effective measure of factual consistency than traditional metrics like FID and CLIPScore. Building upon this evaluation framework, we develop a scene graph feedback pipeline that iteratively refines generated images by identifying and correcting discrepancies between the scene graph and the image. Extensive experiments demonstrate that Scene-Bench provides a more comprehensive and effective evaluation framework compared to existing benchmarks, particularly for complex scene generation. Furthermore, our feedback strategy significantly enhances the factual consistency of image generation models, advancing the field of controllable image generation.

场景图图像生成评估基准反馈机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。