用文字生成合成图像,低成本提升视觉语言模型训练效果
Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models
- 将文本直接渲染成白底图像,打通文本与图像模态鸿沟
- 在7个评测集上优于扩散模型生成的合成图像,提升显著
- 适合数据稀缺场景,尤其适合自动化大规模训练
近期大型视觉语言模型(LVLM)被广泛应用于各类视觉问答任务。然而,实现良好性能通常需大量图像-文本对进行特定任务微调,而这类数据收集成本高昂。本文研究以文本为中心的训练范式:仅使用文本描述,不依赖真实图像,实现低成本数据扩展。相比受隐私和领域稀缺限制的图像,文本更易获取且可由大模型自动改写扩增,人力成本极低。但仅用原始文本训练效果有限,因存在图像-文本模态差距。为此,我们提出文本打印图像(Text-Printed Image, TPI),通过将文本直接渲染在纯白画布上生成合成图像。该方法简单高效,可无缝集成到任意现有LVLM训练流程中,且能准确保留文本语义,而文本到图像模型常出现语义偏差。在四个模型、七个基准上的系统实验表明,TPI在文本为中心训练中表现优于扩散模型生成的合成图像。进一步探索显示其作为低成本数据增强策略具有实际价值。结果表明,文本为中心的训练潜力巨大,为视觉语言模型全自动化数据生成提供了可行路径。
原文摘要 · Abstract (English)
Recent large vision-language models (LVLMs) have been applied to diverse VQA tasks. However, achieving practical performance typically requires task-specific fine-tuning with large numbers of image-text pairs, which are costly to collect. In this work, we study text-centric training, a setting where only textual descriptions are available and no real images are provided, as a paradigm for low-cost data scaling. Unlike images, whose collection is often restricted by privacy constraints and scarcity in niche domains, text is widely available. Moreover, text is easily editable, enabling automatic diversification and expansion with LLMs at minimal human effort. While this offers clear advantages over image collection in terms of scalability and cost, training on raw text without images still yields limited gains on VQA tasks because of the image-text modality gap. To address this issue, we propose a Text-Printed Image (TPI), which generates synthetic images by directly rendering the given textual description on a plain white canvas. This simple rendering projects text into the image modality and can be integrated into arbitrary existing LVLM training pipelines at low cost. Moreover, TPI preserves the semantics of the text, whereas text-to-image models often fail to do. Across four models and seven benchmarks, our systematic experiments show that TPI enables more effective text-centric training than synthetic images generated by a diffusion model. We further explore TPI as a low-cost data-augmentation strategy and demonstrate its practical utility. Overall, our findings highlight the significant potential of text-centric training and, more broadly, chart a path toward fully automated data generation for LVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。