测试大模型对关系、否定和数字逻辑的生成能力，发现表现远低于人类预期。
Relations, Negations, and Numbers: Looking for Logic in Generative Text-to-Image Models
- 用逻辑提示词测试DALL-E 3，评估图像生成是否符合逻辑关系
- 否定和超过3个的数字提示失败率最高，平均匹配率不足50%
- 改进的结构化生成方案反而表现更差，凸显逻辑理解瓶颈
尽管多模态人工智能取得显著进展，但在逻辑运算的可靠应用上，现代AI仍远落后于人类儿童。本文研究了三种逻辑操作：关系、否定与离散数字。我们让178名受试者评估DALL-E 3在逻辑探针提示下的生成图像，发现其人类一致率均未超过50%。否定提示与大于3的数字提示失败最频繁。第四项实验评估了一种基于提示工程与结构化中间表示的“具身扩散”流程，但其表现反而劣于DALL-E 3。通过多个辅助分析，我们量化了关系提示的N-gram频率与图像匹配度的关系、不同否定提示修改策略的成功率，以及涉及整数提示的近似计数能力（比例依赖性）。研究指出，依赖向量语义或语法约束不足的多模态系统存在根本局限，并提出基于认知发展与图像表征的最小改进方案，以缩小规模与结构间的组合鸿沟。所有数据与代码已公开于https://github.com/ColinConwell/T2I-Probology。
原文摘要 · Abstract (English)
Despite remarkable progress in multi-modal AI research, there is a salient domain in which modern AI continues to lag considerably behind even human children: the reliable deployment of logical operators. Here, we examine three forms of logical operators: relations, negations, and discrete numbers. We asked human respondents (N=178 in total) to evaluate images generated by a state-of-the-art image-generating AI (DALL-E 3) prompted with these `logical probes', and find that none reliably produce human agreement scores greater than 50\%. The negation probes and numbers (beyond 3) fail most frequently. In a 4th experiment, we assess a `grounded diffusion' pipeline that leverages targeted prompt engineering and structured intermediate representations for greater compositional control, but find its performance is judged even worse than that of DALL-E 3 across prompts. To provide further clarity on potential sources of success and failure in these text-to-image systems, we supplement our 4 core experiments with multiple auxiliary analyses and schematic diagrams, directly quantifying, for example, the relationship between the N-gram frequency of relational prompts and the average match to generated images; the success rates for 3 different prompt modification strategies in the rendering of negation prompts; and the scalar variability / ratio dependence (`approximate numeracy') of prompts involving integers. We conclude by discussing the limitations inherent to `grounded' multimodal learning systems whose grounding relies heavily on vector-based semantics (e.g. DALL-E 3), or under-specified syntactical constraints (e.g. `grounded diffusion'), and propose minimal modifications (inspired by development, based in imagery) that could help to bridge the lingering compositional gap between scale and structure. All data and code is available at https://github.com/ColinConwell/T2I-Probology
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。