测试大模型能否像专业系统一样判断图表是否符合设计规则。
Do Large Language Models Understand Data Visualization Rules?
- 用自然语言描述可视化规则，让大模型判断图表代码是否违规
- 主流模型能准确识别常见错误，但对微妙感知问题识别能力差
- 小模型在规则转为自然语言后表现提升近1.5倍，适合快速验证
数据可视化规则源于数十年的设计与认知研究，确保图表信息传达的可信性。尽管已有研究显示大语言模型（LLMs）可生成图表或识别误导性图形，但其是否能直接推理并执行可视化规则仍不明确。基于约束的系统如Draco将规则编码为逻辑约束以实现精准自动化检查，但维护符号化编码需专家投入，促使人们探索使用LLMs作为灵活的规则验证器。本文首次采用基于答案集编程（ASP）的硬验证真值，系统评估了LLMs在可视化规则上的表现。我们将Draco的部分约束转化为自然语言表述，并生成包含2000个Vega-Lite规范的受控数据集，每条均标注明确的规则违反情况。评估涵盖检测违规的准确性与提示遵循度（输出是否符合结构化格式）。结果显示，前沿模型在格式遵循上表现优异（Gemma 3 4B / 27B: 100%，GPT-oss 20B: 98%），对常见违规的检测F1最高达0.82；但在细微的感知类规则上表现显著下降（部分类别F1 < 0.15），且对技术性ASP形式的输出处理能力弱。将约束转换为自然语言可使小模型性能提升最多150%。这些发现表明LLMs具备作为语言驱动验证器的潜力，但相较符号求解器仍有明显局限。
原文摘要 · Abstract (English)
Data visualization rules-derived from decades of research in design and perception-ensure trustworthy chart communication. While prior work has shown that large language models (LLMs) can generate charts or flag misleading figures, it remains unclear whether they can reason about and enforce visualization rules directly. Constraint-based systems such as Draco encode these rules as logical constraints for precise automated checks, but maintaining symbolic encodings requires expert effort, motivating the use of LLMs as flexible rule validators. In this paper, we present the first systematic evaluation of LLMs against visualization rules using hard-verification ground truth derived from Answer Set Programming (ASP). We translated a subset of Draco's constraints into natural-language statements and generated a controlled dataset of 2,000 Vega-Lite specifications annotated with explicit rule violations. LLMs were evaluated on both accuracy in detecting violations and prompt adherence, which measures whether outputs follow the required structured format. Results show that frontier models achieve high adherence (Gemma 3 4B / 27B: 100%, GPT-oss 20B: 98%) and reliably detect common violations (F1 up to 0.82),yet performance drops for subtler perceptual rules (F1 < 0.15 for some categories) and for outputs generated from technical ASP formulations.Translating constraints into natural language improved performance by up to 150% for smaller models. These findings demonstrate the potential of LLMs as flexible, language-driven validators while highlighting their current limitations compared to symbolic solvers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。