测试文生图模型能否按物体朝向而非画面坐标理解位置描述。
Can Text-to-Image Models Draw from the Right Frame of Reference?

- 构建新基准FoR-T2I,区分画面视角与物体朝向两种空间描述方式。
- 多数模型在物体朝向描述下准确率比画面视角低41.8个百分点。
- 提出视觉语言模型辅助重写提示,提升无训练场景下的定位准确率。
空间指令遵循已成为文生图生成的关键要求。当方向性表达基于不同参考系时易产生歧义,例如‘左边’可能指画面坐标或物体自身朝向,导致预期布局差异。现有基准虽揭示布局错误,但极少分离模型在非相机视角参考系下的表现。为此,我们提出FoR-T2I,一个包含1200对提示的基准,每对由相机视图(Cam)提示和物体参考系(FoR)提示构成,描述相同目标位置。在22个开源与闭源模型中,平均在FoR提示上的准确率比匹配的Cam提示低41.8%;即使最佳模型也仅达44.3%。这表明当前模型更难处理以物体朝向表达的空间关系。我们进一步分析了不同关系类型与相机视角下的差距,评估多种无需训练的提示优化与反馈策略,并提出一种基于视觉语言模型的提示重写方法,利用视觉反馈选择重写提示,在相同生成预算下将平均FoR准确率从25.0%提升至29.2%。
原文摘要 · Abstract (English)
Spatial instruction following has become a crucial requirement for text-to-image (T2I) generation. A common challenge arises when directional expressions are interpreted under different frames of reference. For example, ``the left of'' may refer to the viewer's image coordinates or to the intrinsic orientation of an object, leading to different expected layouts. Existing T2I benchmarks reveal important layout failures, yet they rarely isolate whether models can follow a specified frame of reference when it differs from camera view. To mitigate this gap, we introduce FoR-T2I, a benchmark for evaluating this distinction with 1,200 prompt pairs built from controlled spatial layouts. In each pair, the camera-view (Cam) prompt states the target relation in camera view, while the frame-of-reference (FoR) prompt describes the same target placement through an oriented anchor object. Across 22 closed-source and open-source T2I models, mean final accuracy is 41.8\% lower on FoR prompts than on matched Cam prompts; even the best-performing model achieves only 44.3\% FoR accuracy. This suggests that current models struggle more when the same layout is described through an object's orientation rather than directly in image coordinates. We further analyze this gap by relation type and camera view, compare several training-free prompting and feedback-based mitigation strategies, and propose a VLM-gated rewriting approach that selects rewritten prompts using visual feedback, improving average FoR accuracy from 25.0\% to 29.2\% under the same generation budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。