arXiv:2510.09887cs.CL2025-10

通过反事实偏好学习提升模型对细微输入的敏感度,显著改善长尾场景下的错误率。

Overconfident and Blind to Details: Fixing Prompt Insensitivity with Abductive Preference Learning

  • 反向优化提示响应概率,利用稀有提示放大改进效果
  • 在VLMBias上准确率从3%提升至44%,增益超14倍
  • 无需修改架构即可适配现有方法,适合关注细节敏感性的研究者

视觉语言模型常忽略语义关键的输入修改,依赖预训练先验。例如,模型会自信声称五条腿的狗有四条腿;在VLMBias基准上,GPT-5.2和Claude Sonnet分别仅达4.6%和0%准确率。现有方法通过构建覆盖罕见输入的数据集来微调策略函数π(y∣x),但提示基线在VLMBias上的提升不足3%,因稀有提示概率过低。为此,本文提出反事实偏好学习,优化反向策略π(x∣y)。理论上,该方法使正向策略改进幅度提升q(y)/p(x)倍,其中p(·)与q(·)分别为提示与响应的边缘概率,对最罕见提示增益最大。进一步证明,对于如DPO等平移不变的成对偏好学习方法,估计π(x∣y)可简化为固定响应的提示交换,无需结构改动。实验表明,反事实偏好学习大幅提升了长尾提示敏感性:在VLMBias上,A-DPO将准确率从3%提升至44%(14倍),超越GPT-5.2(4.6%)及所有闭源VLM(除Gemini 3 Flash);在Inverse-IFEval上,Multi-DPOP达到65%–84%,优于9B规模的GPT-5(73.7%),且保持IFBench性能,而传统DPO导致其下降8%–12%。

原文摘要 · Abstract (English)

Vision and language models frequently ignore semantically critical input edits, defaulting to pretraining priors. For example, models will confidently assert a five-legged dog has four legs; consequently, on the VLMBias benchmark, GPT 5.2 and Claude Sonnet 4.6 achieve only $4.6\%$ and $0\%$ accuracy, respectively. Existing methods address this problem through building up datasets that covers the underrepresented inputs to tune the policy function $π(y \mid x)$, where $x$ and $y$ refer to input prompts and responses, respectively. However, prompting baselines yield gains of under $3\%$ on VLMBias due to the low probability density of rare prompts. To bypass this bottleneck, we propose \emph{abductive preference learning} to optimize the abductive policy $π(x \mid y)$. We prove this amplifies forward policy improvements by a factor of $q(y)/p(x)$, where $p(\cdot)$ and $q(\cdot)$ denote the marginal probabilities of the prompt and response, yielding the largest gains on the rarest prompts. Furthermore, we demonstrate that for translation invariant pairwise preference learning methods, such as DPO, estimating $π(x \mid y)$ reduces to a structural data swap that compares prompts for a fixed response, requiring no architectural changes. Empirically, abductive preference learning delivers large gains on long-tail prompt sensitivity: on VLMBias, A-DPO raises accuracy from $3\%$ to $44\%$ ($14\times$), outperforming GPT-5.2 ($4.6\%$) and all closed-source VLMs except Gemini~3~Flash; on Inverse-IFEval, Multi-DPOP reaches $65$--$84\%$, surpassing GPT-5 ($73.7\%$) at the 9B scale while preserving IFBench, unlike DPO which degrades it by $8$--$12\%$.

视觉语言模型偏好学习长尾敏感性反事实推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。