小模型评估角色图像时易盲目高分,研究提出新指标量化这种盲从行为。
SycoPhantasy: Quantifying Sycophancy and Hallucination in Small Open Weight VLMs for Vision-Language Scoring of Fantasy Characters

- 用'伪装系数'衡量评分与视觉证据的偏离程度
- 450M模型22.3%的评分无视觉依据,远高于7B模型的6.0%
- 提醒在生成内容评估中慎用小型开源视觉模型
视觉语言模型(VLMs)正被用于需要精细图像理解的任务评估,但其在判断图像与文本对齐性方面的可靠性尚未充分研究。本文探究小型开源VLMs在评估图像-文本匹配时是否存在'奉承式'行为:即在缺乏视觉证据的情况下仍给出高分。为此,我们引入'伪装系数'(c),量化模型评分与其视觉证据召回率之间的偏差。我们在包含173,810张AI生成角色肖像与详细文本描述的基准数据集上,评估了6个参数量介于450M至8B的小型开源VLMs。分析显示,模型规模与奉承率存在显著负相关($r = -0.96$, $p = 0.002$),其中最小模型LFM2-VL(450M)在22.3%情况下产生无依据高分,而最大模型LLaVA-1.6(7B)仅为6.0%。该结果对在属性丰富的合成图像评估任务中部署小型开源VLMs作为自动评估器具有直接启示,表明评分与视觉证据间的差距不仅可测量,且后果严重。
原文摘要 · Abstract (English)
Vision-language models (VLMs) are increasingly deployed as evaluators in tasks requiring nuanced image understanding, yet their reliability in scoring alignment between images and text descriptions remains underexplored. We investigate whether small, open-weight VLMs exhibit \emph{sycophantic} behavior when evaluating image-text alignment: assigning high scores without grounding their judgments in visual evidence. To quantify this phenomenon, we introduce the \emph{Bluffing Coefficient} (\bc), a metric that measures the mismatch between a model's score and its evidence recall. We evaluate six open-weight VLMs ranging from 450M to 8B parameters on a benchmark of 173,810 AI-generated character portraits paired with detailed textual descriptions. Our analysis reveals a significant inverse correlation between model size and sycophancy rate ($r = -0.96$, $p = 0.002$), with smaller models exhibiting substantially higher rates of unjustified high scores. The smallest model tested (LFM2-VL, 450M) produced sycophantic evaluations in 22.3\% of cases, compared to 6.0\% for the largest (LLaVA-1.6, 7B). These findings have direct implications for the deployment of small, open-weight VLMs as automated evaluators within attribute-rich, synthetic image evaluation tasks, where the gap between assigned scores and cited visual evidence is both measurable and consequential.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。