用论文内容对科学图表打分,让机器理解图文是否匹配。
SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidence

- 用论文上下文+引用段落+图注,联合训练多模态模型判断图是否支撑论点。
- 在396篇论文上测试,评分误差比最强基线降低59%,准确率达81.6%。
- 适合需要自动化审稿辅助或论文质量评估的研究者使用。
科学图表评估不同于一般图像质量评价:图需清晰可读、忠实支持论文主张,并具有明确的视觉层次。传统图像质量评估模型仅关注感知质量,无法判断图是否服务于科学论证;基于CLIP的方法虽能评估图文对应性,但缺乏论文上下文理解;而直接复用大语言模型或视觉语言模型进行评分,常导致分数集中且难以融合图文证据。我们构建了包含3,857张同行评审会议论文中科学图表的标注数据集,每张图按清晰度、相关性、信息量和结构四维评分。提出SciFigAlign,一种基于论文证据微调的多模态评分器。给定图块、图注、引用段落与轻量论文上下文,通过跨模态注意力与CubeMLP融合,端到端联合优化平滑L1损失与论文内排名铰链损失。在论文级划分下,测试集上宏平均绝对误差达0.3524,论文内成对准确率为81.64%(n=396),相比最佳基线(MAE=0.864)相对误差降低59%。消融实验表明,论文上下文输入、引用上下文去噪与排名监督均至关重要,证明科学图表评估需学习视觉内容与论文证据间的对齐关系,而非仅靠提示工程。
原文摘要 · Abstract (English)
Scientific figure assessment in peer review differs fundamentally from general image quality evaluation: a figure must be visually legible, faithfully support the manuscript's claims, and communicate evidence with a clear visual hierarchy. However, if we apply traditional image assessment methods to scientific figure quality assessment, limitations emerge: classic IQA models capture perceptual quality or aesthetics but cannot judge whether a figure serves the paper's scientific argument; CLIP-based methods assess generic image-text correspondence, yet lack understanding of manuscript context; and zero-shot LLM/VLM judges, when repurposed for figure scoring, often yield overly concentrated scores with limited fusion of visual and textual evidence. We introduce an annotated dataset of 3,857 scientific figures from peer-reviewed conference papers, each rated along four peer-review-oriented dimensions: Clarity, Relevance, Informativeness, and Structure. We propose SciFigAlign, a fine-tuned multimodal scorer that grounds figure quality assessment in manuscript evidence. Given a figure crop, caption, citing paragraphs, and light paper context, SciFigAlign fine-tunes CLIP and SciBERT end-to-end with per-modality cross-attention and CubeMLP fusion, jointly optimizing SmoothL1 regression with a within-paper ranking hinge loss. Under paper-level splits, SciFigAlign achieves a macro MAE of 0.3524 and a within-paper pairwise accuracy of 81.64% on the test set with n = 396, a 59% relative error reduction over the best LLM-as-judge baseline with MAE 0.864. Ablations confirm that manuscript-grounded inputs, citing-context denoising, and ranking supervision are all critical, showing that scientific figure assessment requires learned alignment between visual content and manuscript evidence rather than prompting alone, even with state-of-the-art VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。