用可控错误生成高质量偏好数据，提升视觉语言模型推理能力
VaPR -- Vision-language Preference alignment for Reasoning
- 通过LLM引导编辑生成带特定错误的拒绝响应，保持风格与长度一致
- 在10个基准上平均提升6.5%（LLaVA），推理任务改善显著
- 数据可复用至开源模型，性能接近GPT-4o生成数据
基于AI生成反馈的偏好微调方法（如DPO）在对齐大型视觉语言模型（LVLMs）与人类偏好方面表现良好。然而，现有技术忽视了合成偏好标注中普遍存在的风格和长度偏差等噪声问题。为此，我们提出一种基于LLM引导响应编辑的硬负例生成框架，生成具有目标错误的拒绝响应，同时保持其与接受响应在风格和长度上的相似性。基于该框架，我们构建了包含3万条高质量样本的VaPR数据集，用于微调三种LVLM系列：LLaVA-V1.5、Qwen2VL与Qwen2.5VL（2B-13B规模）。实验表明，使用VaPR微调的模型在10个基准测试中取得显著性能提升，平均增益分别为6.5%（LLaVA）、4.0%（Qwen2VL）和1.5%（Qwen2.5VL），尤其在推理任务中表现突出。缩放分析显示，性能随数据量增加而持续提升，即使在小规模模型上也有效。此外，VaPR降低了模型对二元问题盲目回答“是”的倾向——这是如LLaVA等模型常见的失败模式。最后，我们验证该框架可泛化至开源大模型作为编辑器，使用VaPR-OS训练的模型性能达到使用GPT-4o生成数据训练模型的约99%。数据、模型及代码详见项目主页 https://vap-r.github.io
原文摘要 · Abstract (English)
Preference finetuning methods like Direct Preference Optimization (DPO) with AI-generated feedback have shown promise in aligning Large Vision-Language Models (LVLMs) with human preferences. However, existing techniques overlook the prevalence of noise in synthetic preference annotations in the form of stylistic and length biases. To this end, we introduce a hard-negative response generation framework based on LLM-guided response editing, that produces rejected responses with targeted errors, maintaining stylistic and length similarity to the accepted ones. Using this framework, we develop the VaPR dataset, comprising 30K high-quality samples, to finetune three LVLM families: LLaVA-V1.5, Qwen2VL & Qwen2.5VL (2B-13B sizes). Our VaPR models deliver significant performance improvements across ten benchmarks, achieving average gains of 6.5% (LLaVA), 4.0% (Qwen2VL), and 1.5% (Qwen2.5VL), with notable improvements on reasoning tasks. A scaling analysis shows that performance consistently improves with data size, with LLaVA models benefiting even at smaller scales. Moreover, VaPR reduces the tendency to answer "Yes" in binary questions - addressing a common failure mode in LVLMs like LLaVA. Lastly, we show that the framework generalizes to open-source LLMs as editors, with models trained on VaPR-OS achieving ~99% of the performance of models trained on \name, which is synthesized using GPT-4o. Our data, models, and code can be found on the project page https://vap-r.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。