构建首个图文设计规划基准，测试大模型在创意任务中的规划能力
GraphicBench: A Planning Benchmark for Graphic Design with Language Agents
- 设计四类场景、1079个用户需求的规划基准，支持多专家协作执行
- 六种大模型生成流程可融合显性与隐性设计约束，但执行成功率低
- 适合研究创意自动化、智能体规划与人机协同设计的学者
大语言模型驱动的智能体已能自动化完成许多人类任务。然而，以往研究主要聚焦目标明确的任务，对具有开放目标的创意设计任务中智能体的能力仍缺乏探索。本文提出 GraphicBench，一个覆盖四种设计类型、包含1,079个用户查询和输入图像的规划基准。我们进一步构建 GraphicTown 框架，集成三位设计专家和46种可选动作（工具），用于在网页环境中执行规划工作流。六种大模型实验表明，它们能够生成融合用户查询中显性设计约束与隐性常识约束的工作流。但这些工作流常无法成功执行，主要受限于：(1) 空间关系推理不足；(2) 多专家间的全局依赖协调困难；(3) 每一步骤的动作选择不准确。我们期望 GraphicBench 成为推动大模型智能体在创意设计任务中规划与执行能力发展的关键测试平台。
原文摘要 · Abstract (English)
Large Language Model (LLM)-powered agents have unlocked new possibilities for automating human tasks. While prior work has focused on well-defined tasks with specified goals, the capabilities of agents in creative design tasks with open-ended goals remain underexplored. We introduce GraphicBench, a new planning benchmark for graphic design that covers 1,079 user queries and input images across four design types. We further present GraphicTown, an LLM agent framework with three design experts and 46 actions (tools) to choose from for executing each step of the planned workflows in web environments. Experiments with six LLMs demonstrate their ability to generate workflows that integrate both explicit design constraints from user queries and implicit commonsense constraints. However, these workflows often do not lead to successful execution outcomes, primarily due to challenges in: (1) reasoning about spatial relationships, (2) coordinating global dependencies across experts, and (3) retrieving the most appropriate action per step. We envision GraphicBench as a challenging yet valuable testbed for advancing LLM-agent planning and execution in creative design tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。