Hummingbird让生成图像精准匹配图文上下文,兼顾细节真实与多样性。
Hummingbird: High Fidelity Image Generation via Multimodal Context Alignment
- 通过多模态上下文评估器联合优化语义与细粒度一致性奖励
- 在MME Perception和Bongard HOI上实现最高保真度与多样性平衡
- 适合需要精确场景还原的视觉问答与人物交互任务
尽管扩散模型在生成高质量、多样化的物体中心数据方面表现强大,但现有方法在视觉问答(VQA)和人-物交互(HOI)推理等场景感知任务中仍面临挑战,难以在参考图像与文本提示的多模态上下文中保持场景属性的一致性。为此,我们提出首个基于扩散模型的图像生成器Hummingbird,能够根据多模态上下文生成与参考图像高度多样且高保真的图像,准确保留如物体交互关系和空间位置等场景属性。Hummingbird采用新型多模态上下文评估器,同时优化全局语义与细粒度一致性奖励,确保生成图像在符合文本引导的同时忠实于参考图像的场景结构。作为首个同时处理多模态上下文下多样性与保真度的任务模型,我们引入新基准,涵盖MME Perception和Bongard HOI数据集。实验表明,Hummingbird在多项指标上超越现有方法,验证其在复杂视觉任务中作为鲁棒多模态对齐生成器的潜力。
原文摘要 · Abstract (English)
While diffusion models are powerful in generating high-quality, diverse synthetic data for object-centric tasks, existing methods struggle with scene-aware tasks such as Visual Question Answering (VQA) and Human-Object Interaction (HOI) Reasoning, where it is critical to preserve scene attributes in generated images consistent with a multimodal context, i.e. a reference image with accompanying text guidance query. To address this, we introduce $\textbf{Hummingbird}$, the first diffusion-based image generator which, given a multimodal context, generates highly diverse images w.r.t. the reference image while ensuring high fidelity by accurately preserving scene attributes, such as object interactions and spatial relationships from the text guidance. Hummingbird employs a novel Multimodal Context Evaluator that simultaneously optimizes our formulated Global Semantic and Fine-grained Consistency Rewards to ensure generated images preserve the scene attributes of reference images in relation to the text guidance while maintaining diversity. As the first model to address the task of maintaining both diversity and fidelity given a multimodal context, we introduce a new benchmark formulation incorporating MME Perception and Bongard HOI datasets. Benchmark experiments show Hummingbird outperforms all existing methods by achieving superior fidelity while maintaining diversity, validating Hummingbird's potential as a robust multimodal context-aligned image generator in complex visual tasks. Project page: https://roar-ai.github.io/hummingbird
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。