用分级语气测试视觉语言模型幻觉,发现不同模型响应模式各异。
LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models

- 构建800张合成图像与五级语气提示,分离语气影响
- 发现多个模型在中等语气下幻觉最严重,非单调变化
- 提出双轨评估法,可区分幻觉发生率与具体程度
视觉语言模型在需可靠视觉定位的场景中应用日益广泛,但其在逐步强化的提示语境下的表现仍不清晰。现有幻觉评测多依赖中性提示和二元判断,未揭示幻觉发生频率与强度如何随不同任务类型和语言压力梯度变化。本文提出Ghost-100,一个由程序生成的800张合成图像基准数据集,涵盖文本不可读、时间读取、物体缺失三类任务,每类均基于负向真实原则设计,确保目标在图像中确实不存在或无法确定。每张图像配五组来自五级提示强度框架的提示,仅改变指令力度,使语气成为唯一独立变量。采用双轨评估:规则基H-Rate衡量模型从合理拒绝转向无依据肯定的比例;GPT-4o-mini评判的H-Score在1-5分尺度上刻画幻觉的置信度与具体性。此外发布三阶段自动化验证流程,确认800张中有717张严格合规。对九个开源视觉语言模型评估显示,H-Rate与H-Score在模型族间显著分化,阅读风格与存在检测子集对提示压力反应机制不同,部分模型呈现非单调敏感性,峰值出现在中间语气水平——此类模式被传统聚合指标所掩盖。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are increasingly deployed in settings where reliable visual grounding carries operational consequences, yet their behavior under progressively coercive prompt phrasing remains undercharacterized. Existing hallucination benchmarks predominantly rely on neutral prompts and binary detection, leaving open how both the incidence and the intensity of fabrication respond to graded linguistic pressure across structurally distinct task types. We present Ghost-100, a procedurally constructed benchmark of 800 synthetically generated images spanning eight categories across three task families: text-illegibility, time-reading, and object-absence, each designed under a negative-ground-truth principle that guarantees the queried target is absent, illegible, or indeterminate by construction. Every image is paired with five prompts drawn from a structured 5-Level Prompt Intensity Framework, holding the image and task identity fixed while varying only directive force, so that tone is isolated as the sole independent variable. We adopt a dual-track evaluation protocol: a rule-based H-Rate measuring the proportion of responses in which a model crosses from grounded refusal into unsupported positive commitment, and a GPT-4o-mini-judged H-Score on a 1-5 scale characterizing the confidence and specificity of fabrication once it occurs. We additionally release a three-stage automated validation workflow, which retrospectively confirms 717 of 800 images as strictly compliant. Evaluating nine open-weight VLMs, we find that H-Rate and H-Score dissociate substantially across model families, reading-style and presence-detection subsets respond to prompt pressure in qualitatively different ways, and several models exhibit non-monotonic sensitivity peaking at intermediate tone levels: patterns that aggregate metrics obscure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。