arXiv:2602.00249cs.CVcs.AI2026-02被引 2

SANEval评测文本生成图像的复杂组合能力,能精准诊断失败原因。

SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis

  • 用大模型理解提示,结合开放词汇检测器评估组合一致性。
  • 在6个顶尖模型上验证,与人工评价相关性更高,优于现有基准。
  • 适合研究文本到图像生成、可解释性评估的研究者使用。

文本到图像(T2I)模型虽发展迅速,但在处理多对象、属性和空间关系的复杂提示时仍存在显著瓶颈。现有评测方法受限于封闭词汇表、缺乏细粒度诊断能力,且无法提供可解释的反馈来定位和修复组合性错误。为此,我们提出SANEval(空间、属性与数量评估),构建了一套可扩展的开放词汇组合评测新流程。SANEval利用大语言模型(LLM)深度理解提示,并结合增强型开放词汇目标检测器,实现不受固定词汇限制的组合一致性评估。在六个先进T2I模型上的实验表明,其自动化评估结果更贴近人类判断;该指标在属性绑定、空间关系和数量理解任务中与人工评价的斯皮尔曼等级相关性显著优于现有基准。为推动组合式T2I生成与评测研究,我们将公开SANEval数据集及开源评估工具链。

原文摘要 · Abstract (English)

The rapid progress of text-to-image (T2I) models has unlocked unprecedented creative potential, yet their ability to faithfully render complex prompts involving multiple objects, attributes, and spatial relationships remains a significant bottleneck. Progress is hampered by a lack of adequate evaluation methods; current benchmarks are often restricted to closed-set vocabularies, lack fine-grained diagnostic capabilities, and fail to provide the interpretable feedback necessary to diagnose and remedy specific compositional failures. We solve these challenges by introducing SANEval (Spatial, Attribute, and Numeracy Evaluation), a comprehensive benchmark that establishes a scalable new pipeline for open-vocabulary compositional evaluation. SANEval combines a large language model (LLM) for deep prompt understanding with an LLM-enhanced, open-vocabulary object detector to robustly evaluate compositional adherence, unconstrained by a fixed vocabulary. Through extensive experiments on six state-of-the-art T2I models, we demonstrate that SANEval's automated evaluations provide a more faithful proxy for human assessment; our metric achieves a Spearman's rank correlation with statistically different results than those of existing benchmarks across tasks of attribute binding, spatial relations, and numeracy. To facilitate future research in compositional T2I generation and evaluation, we will release the SANEval dataset and our open-source evaluation pipeline.

文本生成图像组合评估大模型评测开放词汇

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。