为文本生成图像设计更贴近用户使用习惯的离线评估方法
Offline Evaluation of Set-Based Text-to-Image Generation
- 基于用户浏览图像集的行为建模，设计新评估指标
- 在多个数据集上验证，新指标与人类评价高度一致
- 适合关注创意启发场景的生成模型研究者
文本生成图像（TTI）系统常用于创意构思阶段，此时用户需要广泛相关的图像以探索设计空间。现有评估指标多聚焦于分布相似性（如FID），但忽视了用户如何实际浏览和交互图像集合。本文基于排名评估中的成熟方法，构建了一套新的离线评估指标，显式模拟用户对空间排列图像集的浏览行为。所提指标不仅衡量生成图像的相关性，还考虑图像集的多样性与布局合理性。通过在三个不同TTI系统生成的图像网格上开展人类实验，使用MS-COCO captions、Localized Narratives子集及自然场景提示进行分析，结果表明：将评估基准建立在真实用户使用模式之上，是提升基准设计质量的关键且被长期忽视的方向。
原文摘要 · Abstract (English)
Text-to-Image (TTI) systems often support people during ideation, the early stages of a creative process when exposure to a broad set of relevant images can help explore the design space. Since ideation is an important subclass of TTI tasks, understanding how to quantitatively evaluate TTI systems according to how well they support ideation is crucial to promoting research and development for these users. However, existing evaluation metrics for TTI remain focused on distributional similarity metrics like Fréchet Inception Distance (FID). We take an alternative approach and, based on established methods from ranking evaluation, develop TTI evaluation metrics with explicit models of how users browse and interact with sets of spatially arranged generated images. Our proposed offline evaluation metrics for TTI not only capture how relevant generated images are with respect to the user's ideation need but also take into consideration the diversity and arrangement of the set of generated images. We analyze our proposed family of TTI metrics using human studies on image grids generated by three different TTI systems based on subsets of the widely used benchmarks such as MS-COCO captions and Localized Narratives as well as prompts used in naturalistic settings. Our results demonstrate that grounding metrics in how people use systems is an important and understudied area of benchmark design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。