测试大模型能否自动帮普通人生成官方数据图表,发现加引导后效果显著提升。
Are LLMs ready to help non-expert users to make charts of official statistics data?
- 用多维度框架评估大模型从政府数据中自动找数、作图的能力
- 无提示时模型找错数据且图表不规范,加设计指导后表现大幅改善
- 通过自我迭代评估的智能体模式可实现全链路自动化,适合非专业人士使用
在虚假信息泛滥的当下,可靠统计数据的可及性至关重要。国家统计机构提供涵盖广泛主题的结构化数据,但这些信息分散于多个表格中,纯数字形式难以处理,导致实际访问困难。本文探讨当前生成式AI模型是否能帮助非专业用户通过自然语言查询,自动识别正确数据、完成数据处理并生成可视化图表。基于荷兰统计局的公开数据,我们评估了多种大语言模型在数据检索与预处理、代码质量及视觉表达三个维度的表现。结果显示,准确获取和处理数据是最大挑战;多数模型在无明确指导时未能遵循可视化最佳实践。当引入有效的图表设计知识后,模型在可视化评分上显著提升。此外,采用具备迭代自评估能力的智能体方法,在所有评估维度均达到优异表现。研究表明,通过适当的设计引导和反馈机制,大模型已具备实现高精度自动图表生成的能力。
原文摘要 · Abstract (English)
In this time when biased information, deep fakes, and propaganda proliferate, the accessibility of reliable data sources is more important than ever. National statistical institutes provide curated data that contain quantitative information on a wide range of topics. However, that information is typically spread across many tables and the plain numbers may be arduous to process. Hence, this open data may be practically inaccessible. We ask the question "Are current Generative AI models capable of facilitating the identification of the right data and the fully-automatic creation of charts to provide information in visual form, corresponding to user queries?". We present a structured evaluation of recent large language models' (LLMs) capabilities to generate charts from complex data in response to user queries. Working with diverse public data from Statistics Netherlands, we assessed multiple LLMs on their ability to identify relevant data tables, perform necessary manipulations, and generate appropriate visualizations autonomously. We propose a new evaluation framework spanning three dimensions: data retrieval & pre-processing, code quality, and visual representation. Results indicate that locating and processing the correct data represents the most significant challenge. Additionally, LLMs rarely implement visualization best practices without explicit guidance. When supplemented with information about effective chart design, models showed marked improvement in representation scores. Furthermore, an agentic approach with iterative self-evaluation led to excellent performance across all evaluation dimensions. These findings suggest that LLMs' effectiveness for automated chart generation can be enhanced through appropriate scaffolding and feedback mechanisms, and that systems can already reach the necessary accuracy across the three evaluation dimensions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。