通过分步策略提升图文生成数据质量,避免幻觉与错配。
VIVECaption: A Split Approach to Caption Quality Improvement
- 分两步优化:用分层采样构建高质量数据集,再通过微调对齐模型上下文与参数。
- 在开源模型上验证,显著提升图文整体对齐质量。
- 适合需要安全可靠训练数据的企业团队使用。
图文生成模型的训练质量受限于标注质量。当前视觉语言模型生成的描述常存在幻觉、组合推理差和细节理解不足,导致图像与文本不匹配,影响下游模型性能。本文提出 VIVECaption,一种双路径改进方案:首先建立评估指标分类体系,区分通用与实例相关指标;其次提出两阶段方法——基于分层采样的黄金标准数据集构建,以及结合上下文对齐与参数级微调(SFT)的模型对齐策略。实验聚焦结构化描述格式,提升可解析性与下游利用效率。结果表明,在图像描述流水线中引入微调后的字符检测模型,能显著改善图文整体对齐质量。本工作回应企业AI开发对高质量‘无版权’训练数据的迫切需求,提供无需依赖网络爬取内容的实用解决方案。
原文摘要 · Abstract (English)
Caption quality has emerged as a critical bottleneck in training high-quality text-to-image (T2I) and text-to-video (T2V) generative models. While visual language models (VLMs) are commonly deployed to generate captions from visual data, they suffer from hallucinations, poor compositional reasoning, and limited fine-grained understanding, resulting in misaligned image-caption pairs that degrade downstream model performance. This technical report introduces VIVECaption, a systematic two-sided approach to caption quality improvement. We first establish a comprehensive taxonomy of caption evaluation metrics, distinguishing between "universal" and "instance-grounded" metrics, with the ultimate goal of showcasing the use-cases and tradeoffs between different caption quality metrics. We then use this language to describe our two-sided approach to caption quality improvement: (1) a gold-standard dataset creation methodology using stratified sampling and (2) a model alignment strategy encompassing context alignment and parameter-level finetuning using SFT. We demonstrate our methodology on open-source models, focusing on structured caption formats that enable better parsing and downstream utilization. We ultimately show that using a finetuned character detection model in an image captioning pipeline significantly improves holistic image-caption alignment quality. Our work addresses the growing need for high-quality "vegan" training data in enterprise AI development, providing practical solutions for teams seeking to improve caption-image alignment without relying on potentially copyright-protected web-scraped content.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。