构建首个散点图任务基准，评估大模型分析能力
Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks
- 自动生成1.8万张带标注的散点图，覆盖6种数据源和17种设计
- GPT-4和Gemini Flash在聚类计数上准确率超90%，定位任务不足50%
- 提醒避免宽屏或随机配色散点图，影响模型识别效果
AI模型在数据分析与可视化中应用日益广泛，但现有基准很少涵盖散点图特定任务，制约了对模型性能的深入理解。为填补这一空白，我们构建了一个合成且带标注的数据集，包含超过18,000张散点图，来自6种数据生成器和17种图表设计，并基于此建立了一个基准测试。我们评估了OpenAI和Google的专有模型，在五项由聚类边界框、中心坐标和异常值坐标标注的任务上采用N-shot提示。结果显示，OpenAI模型和Gemini 2.5 Flash在聚类计数任务上表现良好，尤其在示例提示下准确率超90%；在异常值识别中，Flash模型精度达65.01%。然而，定位相关任务的表现不佳，精确率和召回率普遍低于或接近50%。图表设计虽影响较小，但仍建议避免使用宽屏比例（16:9和21:9）或随机配色的散点图。补充材料可在https://github.com/feedzai/biy-paper 获取。
原文摘要 · Abstract (English)
AI models are increasingly used for data analysis and visualization, yet benchmarks rarely address scatterplot-specific tasks, limiting insight into performance. To address this gap for one of the most common chart types, we introduce a synthetic, annotated dataset of over 18,000 scatterplots from six data generators and 17 chart designs, and a benchmark based on it. We evaluate proprietary models from OpenAI and Google using N-shot prompting on five distinct tasks derived from annotations of cluster bounding boxes, their center coordinates, and outlier coordinates. OpenAI models and Gemini 2.5 Flash, especially when prompted with examples, are viable options for counting clusters and, in Flash's case, outliers (90%+ Accuracy). However, the results for localization-related tasks are unsatisfactory: Precision and Recall are near or below 50%, except for Flash in outlier identification (65.01%). Furthermore, the impact of chart design on performance appears to be a secondary factor, but it is advisable to avoid scatterplots with wide aspect ratios (16:9 and 21:9) or those colored randomly. Supplementary materials are available at https://github.com/feedzai/biy-paper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。