提出VegaChat框架,解决图表生成评估难题
VegaChat: A Robust Framework for LLM-Based Chart Generation and Assessment
- 设计双指标:Spec Score(无LLM)与Vision Score(多模态评估)
- 在NLV和ChartLLM数据集上人类评分相关性达0.65~0.71
- 支持跨库比较,生成图表无效率趋近于零
基于大语言模型的自然语言转可视化(NL2VIS)系统显著提升了数据可视化的可及性。然而,其进一步应用面临两大挑战:(i) 缺乏标准化评估指标,难以衡量进展与比较方法;(ii) 自然语言描述本身存在歧义,同一查询可能对应多个有效可视化。为此,我们提出VegaChat框架,实现从自然语言生成、验证到评估声明式可视化。提出两种互补指标:Spec Score为确定性指标,无需调用LLM即可衡量指定级别相似性;Vision Score为与库无关的图像级指标,利用多模态LLM评估图表相似性与提示合规性。在NLV语料库及ChartLLM标注子集上评估显示,VegaChat生成无效或空图表的比例接近零,且Spec Score与Vision Score分别与人工判断呈0.65和0.71的皮尔逊相关性,表明所提指标支持一致、跨库比较。代码与评估数据已公开于https://zenodo.org/records/17062309。
原文摘要 · Abstract (English)
Natural-language-to-visualization (NL2VIS) systems based on large language models (LLMs) have substantially improved the accessibility of data visualization. However, their further adoption is hindered by two coupled challenges: (i) the absence of standardized evaluation metrics makes it difficult to assess progress in the field and compare different approaches; and (ii) natural language descriptions are inherently underspecified, so multiple visualizations may be valid for the same query. To address these issues, we introduce VegaChat, a framework for generating, validating, and assessing declarative visualizations from natural language. We propose two complementary metrics: Spec Score, a deterministic metric that measures specification-level similarity without invoking an LLM, and Vision Score, a library-agnostic, image-based metric that leverages a multimodal LLM to assess chart similarity and prompt compliance. We evaluate VegaChat on the NLV Corpus and on the annotated subset of ChartLLM. VegaChat achieves near-zero rates of invalid or empty visualizations, while Spec Score and Vision Score exhibit strong correlation with human judgments (Pearson 0.65 and 0.71, respectively), indicating that the proposed metrics support consistent, cross-library comparison. The code and evaluation artifacts are available at https://zenodo.org/records/17062309.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。