arXiv:2510.00796cs.CVcs.AI2025-10被引 1

用逻辑等价提示词检测文生图模型的语义不一致问题。

MetaLogic: Robustness Evaluation of Text-to-Image Models via Logically Equivalent Prompts

  • 通过语法不同但语义相同的提示对生成图像,对比判断一致性。
  • Flux.dev和DALL-E3的误匹配率分别达59%和71%。
  • 无需真实图像即可发现模型逻辑缺陷,适合模型调试与评估。

近年来,基于扩散模型的文生图(T2I)技术在图像质量上取得显著进步,但仍存在关键缺陷:当提示词发生微小语言变化时,难以保持语义一致性。尽管提示语逻辑等价,生成图像却常出现语义错位或不一致,暴露出模型在推理与泛化上的脆弱性。为此,我们提出MetaLogic——一种无需真实图像作为参考的新型评估框架。该框架利用变异测试,生成语法不同但语义相同的提示对,并通过直接比较生成图像对来识别不一致,从而诊断模型在逻辑理解上的鲁棒性缺陷。不同于传统方法将生成图像与单个提示对比,MetaLogic评估图像对之间的语义等价性,提供一种可扩展、无真值依赖的对齐失败检测方式。它能分类错误类型(如实体缺失、重复、位置错位),并输出反例用于模型调试与优化。我们在多个SOTA T2I模型上评估,发现即使Flux.dev和DALL-E3也分别存在59%和71%的误匹配率。结果表明,MetaLogic不仅高效可扩展,还能有效揭示现有评估指标忽略的细粒度逻辑不一致问题。

原文摘要 · Abstract (English)

Recent advances in text-to-image (T2I) models, especially diffusion-based architectures, have significantly improved the visual quality of generated images. However, these models continue to struggle with a critical limitation: maintaining semantic consistency when input prompts undergo minor linguistic variations. Despite being logically equivalent, such prompt pairs often yield misaligned or semantically inconsistent images, exposing a lack of robustness in reasoning and generalisation. To address this, we propose MetaLogic, a novel evaluation framework that detects T2I misalignment without relying on ground truth images. MetaLogic leverages metamorphic testing, generating image pairs from prompts that differ grammatically but are semantically identical. By directly comparing these image pairs, the framework identifies inconsistencies that signal failures in preserving the intended meaning, effectively diagnosing robustness issues in the model's logic understanding. Unlike existing evaluation methods that compare a generated image to a single prompt, MetaLogic evaluates semantic equivalence between paired images, offering a scalable, ground-truth-free approach to identifying alignment failures. It categorises these alignment errors (e.g., entity omission, duplication, positional misalignment) and surfaces counterexamples that can be used for model debugging and refinement. We evaluate MetaLogic across multiple state-of-the-art T2I models and reveal consistent robustness failures across a range of logical constructs. We find that even the SOTA text-to-image models like Flux.dev and DALLE-3 demonstrate a 59 percent and 71 percent misalignment rate, respectively. Our results show that MetaLogic is not only efficient and scalable, but also effective in uncovering fine-grained logical inconsistencies that are overlooked by existing evaluation metrics.

文生图模型评估逻辑一致性鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。