用角色扮演生成更丰富的图文数据，提升模型性能。
Role-SynthCLIP: A Role Play Driven Diverse Synthetic Data Approach
- 让大模型从不同角色视角生成图文描述，增强语义多样性。
- 仅用100万对数据训练的模型在COCO上达64.1%准确率，超现有方法2.8个百分点。
- 适合需要高质量合成数据的多模态研究者使用。
对比语言-图像预训练（CLIP）模型的效果高度依赖于训练数据的语义多样性和质量。然而，现有合成数据生成方法主要关注数据量增加，常导致语义多样性不足和描述冗余或浅显。为此，我们提出Role-SynthCLIP，一种基于多视角角色扮演提示（如构图分析师、图像上下文解读者）的新型数据合成框架，引导多模态大模型从不同视角生成语义丰富的图文描述。该机制提升了合成对的语义多样性与细粒度图文对齐能力，增强了描述的表现力与准确性，同时保持图像-文本对总数不变。实验表明，该方法高效且有效：仅用100万对角色合成数据训练的CLIP-B/16模型，在MS COCO验证集上达到64.1% Recall@1，优于现有最佳合成数据基线（500万对）2.8个百分点。代码与模型已开源：https://github.com/huangfu170/Role-SynthCLIP。
原文摘要 · Abstract (English)
The effectiveness of Contrastive Language-Image Pre-training (CLIP) models critically depends on the semantic diversity and quality of their training data. However, while existing synthetic data generation methods primarily focus on increasing data volume, such emphasis often leads to limited semantic diversity and redundant or shallow captions. To address this limitation, we propose Role-SynthCLIP, a novel data synthesis framework that leverages multi-perspective role-playing prompts (e.g., a compositional analyst, an interpreter of image context) to guide Multimodal Large Language Models (MLLMs) in generating semantically diverse captions from distinct viewpoints. This mechanism enhances the semantic diversity and fine-grained image-text alignment of synthetic pairs, thereby improving caption expressiveness and accuracy while keeping the total number of image-text pairs unchanged. Experimental results demonstrate the effectiveness and efficiency of our method. A CLIP-B/16 model trained on only 1 million Role-SynthCLIP pairs achieves a Recall@1 of 64.1% on the MS COCO validation set, surpassing the best existing synthetic data baseline (trained on 5M pairs) by 2.8 percentage points. The code and trained models are released at https://github.com/huangfu170/Role-SynthCLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。