用鼠标绘图任务评估大模型的空间推理与界面交互能力，提升AI代理可信度。
DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks
- 通过鼠标操作的绘图任务测试模型空间推理与界面交互能力。
- 4个顶尖模型在1000次测试中平均92.8%完美完成，外部反馈提升3.2%-32.8%。
- 明确指令比任务复杂度更关键，透明评估可增强可信度，适合可信AI研究者使用。
随着智能体型AI系统日益自主运行，通过可验证评估建立信任变得至关重要。现有基准缺乏透明度和可审计性，难以评估智能体行为是否可靠。我们提出DrawingBench，一个基于空间推理任务的可验证评估框架，要求生成一系列低层级的GUI操作。不同于黑箱评估，DrawingBench提供透明、规则化的评分：8项客观标准实现可复现打分，动作级检查使利益相关方可审计智能体行为。框架包含250个多样化提示，覆盖20个类别和4个难度等级，具备确定性评估指标，并通过多轮反馈实现外部监督，支持人类对智能体优化进行控制。在1000次测试中评估四个领先大模型（Claude-4 Sonnet、GPT-4.1、GPT-4.1-mini、Gemini-2.5 Flash），发现模型在结构化外部反馈下表现显著提升（平均+3.2%，最高+32.8%），但在工具状态管理与长程规划上存在系统性错误。值得注意的是，指令清晰度比任务复杂度更重要——当给出明确可验证标准时，模型达成100%完美表现。结果表明，透明评估框架有助于建立对智能体系统的信任，外部监督比自我修正更可靠。开源框架为可信智能体评估提供模板。代码与数据：https://github.com/hyunjun1121/DrawingBench
原文摘要 · Abstract (English)
As agentic AI systems increasingly operate autonomously, establishing trust through verifiable evaluation becomes critical. Yet existing benchmarks lack the transparency and auditability needed to assess whether agents behave reliably. We present DrawingBench, a verification framework for evaluating the trustworthiness of agentic LLMs through spatial reasoning tasks that require generating sequences of low-level GUI actions. Unlike opaque evaluations, DrawingBench provides transparent, rule-based assessment: 8 objective criteria enable reproducible scoring, while action-level inspection allows stakeholders to audit agent behavior. Our framework comprises 250 diverse prompts across 20 categories and 4 difficulty levels, deterministic evaluation metrics, and an external oversight mechanism through multi-turn feedback that enables human control over agent refinement. Evaluating four state-of-the-art LLMs (Claude-4 Sonnet, GPT-4.1, GPT-4.1-mini, Gemini-2.5 Flash) across 1,000 tests, we establish both capabilities and limitations: models achieved 92.8% perfect performance with structured external feedback driving significant improvements (average +3.2%, up to +32.8% for complex scenes), but systematic error patterns emerged in tool state management and long-horizon planning. Notably, specification clarity proved more important than task complexity -- models achieved 100% perfect performance when given explicit, verifiable criteria. These findings demonstrate that transparent evaluation frameworks can establish trust in agentic systems, with external oversight proving more reliable than self-correction for guiding agent behavior. Our open-source framework provides a template for trustworthy agent assessment. Code and data: https://github.com/hyunjun1121/DrawingBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。