用多个大模型协作生成科学图表描述，质量超过人工撰写的标题。
Multi-LLM Collaborative Caption Generation in Scientific Documents
- 分三步：先筛低质数据，再多模型生成候选，最后选优并修正
- 生成的标题在人类评估中表现优于人工撰写，准确率显著提升
- 适合需要高质量图表描述的科研人员和论文自动化工具开发者
科学图表标题生成是一项复杂任务，需生成与上下文相符的视觉内容描述。现有方法常因信息不全，将任务简化为图像到文本或文本摘要问题，难以生成高质量标题。此外，来自arXiv的训练数据存在大量低质标题，影响大语言模型（LLM）的训练效果。本文提出多大模型协作的图表标题生成框架MLBCAP，通过专业化分工解决上述问题：（质量评估）利用多模态大模型筛选训练数据，剔除低质标题；（多样生成）对多个大模型进行微调/提示，生成候选标题；（判断优化）由领先的大模型从候选中选出最优标题，并修正剩余错误。人类评估显示，本方法生成的标题在信息量上优于人工撰写，证明其有效性。代码已开源。
原文摘要 · Abstract (English)
Scientific figure captioning is a complex task that requires generating contextually appropriate descriptions of visual content. However, existing methods often fall short by utilizing incomplete information, treating the task solely as either an image-to-text or text summarization problem. This limitation hinders the generation of high-quality captions that fully capture the necessary details. Moreover, existing data sourced from arXiv papers contain low-quality captions, posing significant challenges for training large language models (LLMs). In this paper, we introduce a framework called Multi-LLM Collaborative Figure Caption Generation (MLBCAP) to address these challenges by leveraging specialized LLMs for distinct sub-tasks. Our approach unfolds in three key modules: (Quality Assessment) We utilize multimodal LLMs to assess the quality of training data, enabling the filtration of low-quality captions. (Diverse Caption Generation) We then employ a strategy of fine-tuning/prompting multiple LLMs on the captioning task to generate candidate captions. (Judgment) Lastly, we prompt a prominent LLM to select the highest quality caption from the candidates, followed by refining any remaining inaccuracies. Human evaluations demonstrate that informative captions produced by our approach rank better than human-written captions, highlighting its effectiveness. Our code is available at https://github.com/teamreboott/MLBCAP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。