让视觉语言模型生成超长且精准的文本，突破千字瓶颈。
LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models
- 用2.2万条长文本指令数据微调模型，解决输出短问题。
- 通过分段迭代优化，实现3000词以上高保真生成。
- 适合需要长文本生成的科研、创作与复杂图文任务。
现有大型视觉语言模型（LVLM）可处理长达128k视觉与文本标记的输入，但在生成超过1,000字的连贯输出时仍存在困难。我们发现主要原因是监督微调（SFT）阶段缺乏长输出样本。为此，我们构建了LongWriter-V-22k数据集，包含22,158个样本，每个样本含多张图像、指令及0至10,000字的对应输出。为保持生成内容对输入图像的高保真度，我们在SFT模型上采用直接偏好优化（DPO）。由于收集3,000词以上输出的人工反馈成本高昂，我们提出IterDPO，将长输出分段并迭代修正，形成偏好对。此外，我们开发了MMLongBench-Write基准，涵盖六个任务以评估模型的长文本生成能力。我们训练的7B参数模型在该基准上表现优异，超越更大的专有模型如GPT-4o。代码与数据：https://github.com/THU-KEG/LongWriter-V
原文摘要 · Abstract (English)
Existing Large Vision-Language Models (LVLMs) can process inputs with context lengths up to 128k visual and text tokens, yet they struggle to generate coherent outputs beyond 1,000 words. We find that the primary limitation is the absence of long output examples during supervised fine-tuning (SFT). To tackle this issue, we introduce LongWriter-V-22k, a SFT dataset comprising 22,158 examples, each with multiple input images, an instruction, and corresponding outputs ranging from 0 to 10,000 words. Moreover, to achieve long outputs that maintain high-fidelity to the input images, we employ Direct Preference Optimization (DPO) to the SFT model. Given the high cost of collecting human feedback for lengthy outputs (e.g., 3,000 words), we propose IterDPO, which breaks long outputs into segments and uses iterative corrections to form preference pairs with the original outputs. Additionally, we develop MMLongBench-Write, a benchmark featuring six tasks to evaluate the long-generation capabilities of VLMs. Our 7B parameter model, trained with LongWriter-V-22k and IterDPO, achieves impressive performance on this benchmark, outperforming larger proprietary models like GPT-4o. Code and data: https://github.com/THU-KEG/LongWriter-V
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。