用20亿参数小模型实现高效图表理解评估，适合资源有限场景
Deploying Tiny LVLM Judges for Real-World Evaluation of Chart Models: Lessons Learned and Best Practices
- 将多评价标准合并为单一指令，提升评估效率
- 20亿参数模型经微调后在图表任务上表现媲美70亿大模型
- 方法可跨数据集迁移，适合实际部署与低成本评测
70亿参数的大型视觉语言模型（LVLM）在图表理解任务中已展现作为自动评判器的潜力。然而，20亿参数以下的极小模型仍难以胜任评判任务，限制了其在资源受限场景中的应用。为此，我们提出两种低成本评估方案：(i) 多标准提示法，将多个评价维度整合为单个查询；(ii) 领域自适应迁移学习，利用合成评判数据对20亿参数的LVLM进行微调，构建出名为ChartJudge的小型专用评判模型。实验表明，多标准提示暴露了模型鲁棒性缺陷，导致70亿模型性能显著下降，包括专用评判模型LLaVA-Critic。此外，我们发现该小型模型能有效实现跨数据集的知识迁移，具备更强的专属性。对不同图表类型与查询复杂度的细粒度分析揭示了模型规模、提示设计与可迁移性之间的权衡关系，为可扩展、低成本的图表推理评估提供了实用方案。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) with only 7B parameters have shown promise as automated judges in chart comprehension tasks. However, tiny models (<=2B parameters) still perform poorly as judges, limiting their real-world use in resource-constrained settings. To address this, we propose two approaches to ensure cost-efficient evaluation: (i) multi-criteria prompting, which combines separate evaluation criteria into a single query, and (ii) domain-adaptive transfer learning, in which we fine-tune a 2B-parameter LVLM on synthetic judgments in a chart dataset to create the ChartJudge. Experiments show that multi-criteria prompting exposes robustness gaps, which led to a huge drop in performance for 7B models, including specialized LVLM judges like LLaVA-Critic. In addition, we find that our tiny LVLM (ChartJudge) can effectively transfer knowledge from one dataset to another to make it a more specialized model. Our fine-grained analysis across chart types and query complexities offers actionable insights into trade-offs between model size, prompt design, and transferability, enabling scalable, low-cost evaluation for chart reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。