解决视觉语言模型对齐中的概率坍缩问题,提升生成真实性。
ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints
- 引入非对称约束优化,动态调整拒绝样本梯度
- 有效防止视觉线索丢失,减少幻觉现象
- 适用于多模态任务,适合追求高真实性的模型优化
尽管直接偏好优化(DPO)已成为对齐大视觉语言模型(LVLMs)的主流方法,但其存在概率坍缩问题,即优选和次优响应的概率同时下降。在多模态场景中,优选概率的衰减——我们称之为视觉锚点坍缩——导致模型放弃视觉证据,转而依赖语言先验,引发严重幻觉。为此,我们提出非对称约束偏好优化(ACPO),一种与模态无关的对齐机制,通过动态、目标导向的缩放策略作用于偏好优化过程。ACPO仅对拒绝项施加复杂度感知的缩放系数,不对称地抑制其梯度流,同时保持优选分布作为稳定的梯度参考。实验表明,ACPO能有效逆转标准DPO的优选奖励退化,在InternVL模型上显著优于基线,在幻觉评测集(HallusionBench、MM-IFEval)和通用榜单(MMBench、MMStar、OCRBenchV2)上均表现更优,同时提升整体能力。
原文摘要 · Abstract (English)
While Direct Preference Optimization (DPO) has become the de facto approach for aligning Large Vision-Language Models (LVLMs), it suffers from Likelihood Displacement, where the probability of both chosen and rejected responses collapses. This optimization flaw is especially detrimental in multimodal settings: the erosion of chosen likelihoods -- a failure we term Visual Anchor Collapse -- causes models to abandon visual evidence for strong language priors, precipitating significant hallucinations. To address this, we propose Asymmetric Constrained Preference Optimization (ACPO), a modality-agnostic alignment mechanism that applies dynamic, target-oriented scaling to preference optimization. ACPO derives a complexity-aware scaling coefficient applied exclusively to the rejected reward, asymmetrically suppressing the gradient flow on the rejected term while preserving the chosen distribution as a gradient-stable reference. While fundamentally a general-purpose objective, breaking this gradient symmetry is crucial for multimodal tasks, as it mitigates the suppression of visual tokens by language priors. Experiments on InternVL models demonstrate that ACPO effectively reverses the chosen-reward degradation of standard DPO. By halting Visual Anchor Collapse, ACPO generally outperforms baselines on hallucination benchmarks (HallusionBench, MM-IFEval) and general leaderboards (MMBench, MMStar, OCRBenchV2) while driving concurrent improvements in general capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。