arXiv:2506.15583cs.CL2025-06EMNLP被引 1

解决多句描述的图文图谱构建难题,提升视觉语言模型表现

DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement

论文配图:DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement
图 1 · 摘自论文原文
  • 通过迭代图优化机制,从零开始生成并逐步修正跨句语义图
  • 在400个专家标注数据上实现30%以上SPICE提升,推理速度比GPT-4o快86倍
  • 适合需要高效、准确处理复杂多句图文描述的研究与应用

视觉语言模型生成包含多句的连贯视觉描述,对传统单句图文图谱解析器构成挑战。现有方法通常合并句子级输出,易遗漏跨句指代等现象,导致图谱碎片化,影响下游任务性能。本文提出新任务‘篇章级文本图谱解析’(DiscoSG),并发布包含400个专家标注和8,430个合成样本的DiscoSG-DS数据集,每条描述平均9句话,每个图谱三倍于现有数据集的三元组数量。微调GPT-4o在该数据集上获得超过40%的SPICE提升,但存在推理成本高和授权限制问题。较小的开源模型(如Flan-T5)在简单图上表现良好,但在复杂图上性能下降。为此,我们提出DiscoSG-Refiner——一个轻量级开源解析器,通过种子图生成与新型图编辑模型的迭代优化,在保持低延迟的同时,实现30%的SPICE提升,且在篇章级描述评估与幻觉检测等下游任务中优于其他开源方案。代码与数据见https://github.com/ShaoqLin/DiscoSG。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) generate discourse-level, multi-sentence visual descriptions, challenging text scene graph parsers built for single-sentence caption-to-graph mapping. Current approaches typically merge sentence-level parsing outputs for discourse input, often missing phenomena like cross-sentence coreference, resulting in fragmented graphs and degraded downstream VLM task performance. We introduce a new task, Discourse-level text Scene Graph parsing (DiscoSG), and release DiscoSG-DS, a dataset of 400 expert-annotated and 8,430 synthesised multi-sentence caption-graph pairs. Each caption averages 9 sentences, and each graph contains at least 3 times more triples than those in existing datasets. Fine-tuning GPT-4o on DiscoSG-DS yields over 40% higher SPICE metric than the best sentence-merging baseline. However, its high inference cost and licensing restrict open-source use. Smaller fine-tuned open-source models (e.g., Flan-T5) perform well on simpler graphs yet degrade on denser, more complex graphs. To bridge this gap, we introduce DiscoSG-Refiner, a lightweight open-source parser that drafts a seed graph and iteratively refines it with a novel learned graph-editing model, achieving 30% higher SPICE than the baseline while delivering 86 times faster inference than GPT-4o. It generalises from simple to dense graphs, thereby consistently improving downstream VLM tasks, including discourse-level caption evaluation and hallucination detection, outperforming alternative open-source parsers. Code and data are available at https://github.com/ShaoqLin/DiscoSG .

图文理解图谱解析多句生成开源模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。