用偏好学习提升图像描述生成，不依赖准确标注。
MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation
- 通过多维度偏好优化，无需精确标签即可训练模型。
- 在202k样本数据上，优于DPO和SFT方法。
- 适合无障碍设计与多模态模型研究者使用。
图像替代文本（alt-text）生成任务旨在为图像提供简洁、上下文相关的描述，帮助视障用户获取网络图片信息。尽管大型视觉语言模型具备强大能力，但其表现受限于用户生成标注的噪声、标准不一以及对上下文信息的敏感度不足。以往基于监督微调（SFT）的方法因依赖准确的目标标注而效果不佳，而这些标注往往存在缺陷。为此，我们提出多面跨模态直接偏好优化（MCM-DPO），通过学习在偏好对中识别更优选项来改进生成质量，无需精确标注。MCM-DPO 在单个、成对及多偏好维度上优化文本、视觉与跨模态因素。针对高质量标注与偏好标注数据稀缺的问题，我们构建了两个大规模高质量数据集 TAlt 与 PAlt，分别来自 Twitter 与 Pinterest，包含 202,000 个标注样本和 18,000 个偏好对，涵盖多样化偏好维度，以支持该领域进一步研究。实验表明，所提 MCM-DPO 方法在多个指标上持续优于 DPO 和 SFT，建立新的性能基准。代码与数据已开源：https://github.com/LVUGAI/MCM-DPO。
原文摘要 · Abstract (English)
The alt-text generation task produces concise, context-relevant descriptions of images, enabling blind and low-vision users to access online images. Despite the capabilities of large vision-language models, alt-text generation performance remains limited due to noisy user annotations, inconsistent standards, and MLLMs' insensitivity to contextual information. Previous efforts to fine-tune MLLMs using supervised fine-tuning (SFT) have struggled, as SFT relies on accurate target annotations, which are often flawed in user-generated alt-text. To address this, we propose Multi-faceted Cross-modal Direct Preference Optimization (MCM-DPO), which improves alt-text generation by learning to identify better options in preference pairs without requiring precise annotations. MCM-DPO optimizes preferences across single, paired, and multi-preference dimensions, covering textual, visual, and cross-modal factors. In light of the scarcity of high-quality annotated and preference-labeled datasets for alt-text, we constructed two large-scale, high-quality datasets named TAlt and PAlt, sourced from Twitter and Pinterest. These datasets include 202k annotated alt-text samples and 18k preference pairs that cover diverse preference dimensions, aiming to support further research in this domain. Experimental results show that our proposed MCM-DPO method consistently outperforms both DPO and SFT, establishing a new state of the art in alt-text generation. We release the code and data here: https://github.com/LVUGAI/MCM-DPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。