让图文任务计划更一致连贯,一步接一步优化生成。
Enhance Multimodal Consistency and Coherence for Text-Image Plan Generation
- 分步迭代生成图文计划,用视觉信息反哺文本优化。
- 在1100个日常任务上验证,图文一致性提升显著。
- 适合想改进图文生成质量的研究者与开发者。
人们通过包含文字和图像的多种媒体了解日常任务计划。然而,以往研究多集中于大语言模型生成纯文本计划的能力,而大规模模型在生成图文计划方面的潜力尚未充分探索。高质量图文计划面临两大挑战:跨模态的一致性对齐以及视觉步骤间的连贯性。为此,我们提出一种新型分步生成与优化框架。每轮迭代中,该框架(1)基于历史预测生成下一文本步骤;(2)编辑前一视觉步骤得到新视觉步骤;(3)提取类PDDL的视觉信息;(4)利用提取的视觉信息优化文本草稿。第(2)和(4)步产出的图文步骤将作为下一轮输入。该方法可无缝集成至Mistral-7B、Gemini-1.5、GPT-4o等主流模型。为评估效果,我们构建了包含1100个任务及其图文解决方案的新基准,覆盖11类日常主题,并设计了一套新指标用于衡量图文计划的多模态一致性与连贯性。大量实验表明,该方法在多种基线模型上均表现优异。代码与数据已开源。
原文摘要 · Abstract (English)
People get informed of a daily task plan through diverse media involving both texts and images. However, most prior research only focuses on LLM's capability of textual plan generation. The potential of large-scale models in providing text-image plans remains understudied. Generating high-quality text-image plans faces two main challenges: ensuring consistent alignment between two modalities and keeping coherence among visual steps. To address these challenges, we propose a novel framework that generates and refines text-image plans step-by-step. At each iteration, our framework (1) drafts the next textual step based on the prediction history; (2) edits the last visual step to obtain the next one; (3) extracts PDDL-like visual information; and (4) refines the draft with the extracted visual information. The textual and visual step produced in stage (4) and (2) will then serve as inputs for the next iteration. Our approach offers a plug-and-play improvement to various backbone models, such as Mistral-7B, Gemini-1.5, and GPT-4o. To evaluate the effectiveness of our approach, we collect a new benchmark consisting of 1,100 tasks and their text-image pair solutions covering 11 daily topics. We also design and validate a new set of metrics to evaluate the multimodal consistency and coherence in text-image plans. Extensive experiment results show the effectiveness of our approach on a range of backbone models against competitive baselines. Our code and data are available at https://github.com/psunlpgroup/MPlanner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。