评测多模态大模型读取工程图表的能力，精准提取量化数值。
PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading
- 用已知参数生成带真实答案的图表，实现可复现评估。
- 引入中间检查点，定位模型在读取频段、峰值等子任务中的失败。
- 发现频率域任务仍难，如频谱分析准确率不足23%。
我们提出 PlotChain，一个确定性的、基于生成器的基准，用于评估多模态大语言模型（MLLM）在工程图表阅读上的表现——即从经典图表（如伯德图、FFT图、阶跃响应、应力-应变图、泵特性曲线）中恢复定量数值，而非仅做文字识别或自由描述。PlotChain 包含 15 种图表类型，共 450 张渲染图（每类 30 张），每张图均由已知参数生成，并配有由生成过程直接计算出的精确真实值。核心贡献是基于检查点的诊断性评估：除最终目标外，每个样本还包含中间 'cp_' 字段，用于隔离子技能（如读取截止频率或峰值幅值），并实现故障定位。我们在标准化、确定性协议下（温度=0，严格仅输出JSON数值格式）评估了四个最先进的 MLLM，使用按字段设定的容差策略衡量预测精度，该策略反映人类读图的精确度。在 'plotread' 容差政策下，顶尖模型整体字段通过率分别为：Gemini 2.5 Pro 80.42%，GPT-4.1 79.84%，Claude Sonnet 4.5 78.21%；GPT-4o 为 61.59%。尽管多数任务表现良好，但频率域任务依然脆弱：带通响应准确率 ≤23%，FFT 谱图仍具挑战性。我们公开生成器、数据集、原始模型输出、评分代码和校验文件，支持完全可复现运行及未来基于不同容差策略的回溯评分。
原文摘要 · Abstract (English)
We present PlotChain, a deterministic, generator-based benchmark for evaluating multimodal large language models (MLLMs) on engineering plot reading-recovering quantitative values from classic plots (e.g., Bode/FFT, step response, stress-strain, pump curves) rather than OCR-only extraction or free-form captioning. PlotChain contains 15 plot families with 450 rendered plots (30 per family), where every item is produced from known parameters and paired with exact ground truth computed directly from the generating process. A central contribution is checkpoint-based diagnostic evaluation: in addition to final targets, each item includes intermediate 'cp_' fields that isolate sub-skills (e.g., reading cutoff frequency or peak magnitude) and enable failure localization within a plot family. We evaluate four state-of-the-art MLLMs under a standardized, deterministic protocol (temperature = 0 and a strict JSON-only numeric output schema) and score predictions using per-field tolerances designed to reflect human plot-reading precision. Under the 'plotread' tolerance policy, the top models achieve 80.42% (Gemini 2.5 Pro), 79.84% (GPT-4.1), and 78.21% (Claude Sonnet 4.5) overall field-level pass rates, while GPT-4o trails at 61.59%. Despite strong performance on many families, frequency-domain tasks remain brittle: bandpass response stays low (<= 23%), and FFT spectrum remains challenging. We release the generator, dataset, raw model outputs, scoring code, and manifests with checksums to support fully reproducible runs and retrospective rescoring under alternative tolerance policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。