用场景图生成高质量合成数据,提升视觉生成模型的组合泛化能力。
Generate Any Scene: Scene Graph Driven Data Synthesis for Visual Generation Training
- 基于物体、属性和关系的结构化分类体系,自动生成复杂场景图。
- 仅用800条合成数据使Stable Diffusion性能提升10%,在复杂概念生成上超越基线。
- 适用于需要高语义对齐的场景,如内容审核、模型蒸馏与自动评估。
当前文本到图像生成模型在视觉保真度上表现优异,但在组合泛化与语义对齐方面仍存在不足。现有数据集噪声大且弱组合性,限制了模型对复杂场景的理解,而密集高质量标注的可扩展解决方案仍具挑战。我们提出Generate Any Scene,一个系统化枚举可能视觉场景的场景图数据引擎。该引擎从对象、属性、关系的结构化分类体系中动态构建不同复杂度的场景图;给定场景图后,可生成用于文本到图像/视频生成的描述文本,也可生成视觉问题答案集,实现模型生成结果的自动评估与奖励建模。利用该引擎,我们设计自迭代优化框架,使Stable Diffusion v1.5在CC3M上平均提升4%;提出蒸馏算法,仅用不到800条合成描述微调Stable Diffusion v1.5,使其在组合与难概念生成上的TIFA得分提升10%;构建低成本奖励模型,结合GRPO算法微调SimpleAR-0.5B-SFT,在DPG-Bench上比基于CLIP的方法高出+5%。最后,将该方法应用于内容审核任务,通过合成数据训练模型识别复杂违规案例。
原文摘要 · Abstract (English)
Recent advances in text-to-vision generation excel in visual fidelity but struggle with compositional generalization and semantic alignment. Existing datasets are noisy and weakly compositional, limiting models' understanding of complex scenes, while scalable solutions for dense, high-quality annotations remain a challenge. We introduce Generate Any Scene, a data engine that systematically enumerates scene graphs representing the combinatorial array of possible visual scenes. Generate Any Scene dynamically constructs scene graphs of varying complexity from a structured taxonomy of objects, attributes, and relations. Given a sampled scene graph, Generate Any Scene translates it into a caption for text-to-image or text-to-video generation; it also translates it into a set of visual question answers that allow automatic evaluation and reward modeling of semantic alignment. Using Generate Any Scene, we first design a self-improving framework where models iteratively enhance their performance using generated data. Stable Diffusion v1.5 achieves an average 4% improvement over baselines and surpassing fine-tuning on CC3M. Second, we also design a distillation algorithm to transfer specific strengths from proprietary models to their open-source counterparts. Using fewer than 800 synthetic captions, we fine-tune Stable Diffusion v1.5 and achieve a 10% increase in TIFA score on compositional and hard concept generation. Third, we create a reward model to align model generation with semantic accuracy at a low cost. Using GRPO algorithm, we fine-tune SimpleAR-0.5B-SFT and surpass CLIP-based methods by +5% on DPG-Bench. Finally, we apply these ideas to the downstream task of content moderation where we train models to identify challenging cases by learning from synthetic data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。