arXiv:2608.10835cs.CVcs.LG2026-08

轻量级检测大模型生成中的幻觉,定位到具体词元。

UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations

论文配图:UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations
图 1 · 摘自论文原文
  • 构建图像块、查询词元与生成词元的有向图,融合空间、关系与顺序信息。
  • 在多个大模型上实现顶尖的词元级幻觉检测,生成时可降低55%物体幻觉。
  • 无需微调模型,适合部署于各类视觉语言模型,支持实时检测与重采样。

大型视觉语言模型(LVLMs)虽具备出色的视觉推理与对话能力,但常产生与图像输入无关的幻觉内容。有效缓解需实现词元级别的幻觉定位,以实现精准干预而不丢弃整段回答。现有检测方法需昂贵的全模型微调,依赖外部验证器而忽略生成过程,或仅提取孤立特征与手工统计量,丢失空间、序列与关系结构。本文提出 extbf{UniProbe},一种轻量、统一、可学习的检测器,通过单次前向传播建模冻结的LVLM的异构计算轨迹。它构建图像块、查询词元与生成词元间的有向图,注意力权重表征其关系。采用交替的结构感知模块:图神经网络捕捉关系证据,视觉变换器处理二维视觉几何,门控循环单元建模生成顺序。模块交织使空间、关系与顺序证据全程交互。进一步设计流式版本用于幻觉感知解码,在生成中实时检测并重采样幻觉词元,并引入自适应策略使检测器与模型自身生成对齐。在多种LVLM骨干网络上,UniProbe达到当前最优的词元级与物体幻觉检测性能。解码阶段,幻觉减少最高达55%,延迟仅为标准生成的1.06倍。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input. Effective mitigation requires token-level localization, enabling targeted intervention without discarding the entire response. Existing detectors require expensive full-model fine-tuning, rely on external verifiers that ignore the model's generation process, or reduce internal signals to isolated features and hand-crafted statistics, discarding spatial, sequential, and relational structure. We introduce \textbf{UniProbe}, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass. UniProbe constructs a directed graph over image patches, query tokens, and generated tokens, with attention weights encoding their relations. It processes this trace with alternating structure-aware modules: a GNN for relational evidence, a ViT for 2-D visual geometry, and a GRU for response order. Interleaving them allows spatial, relational, and sequential evidence to interact throughout the detector. We further develop a streaming variant for hallucination-aware decoding, which detects and resamples hallucinated tokens during generation, and a self-adaptation strategy aligning the detector with the LVLM's own generations. Across diverse LVLM backbones, UniProbe achieves state-of-the-art token-level and object-hallucination detection. During decoding, it reduces object hallucinations by up to 55\% at $1.06\times$ the latency of standard generation.

幻觉检测视觉语言模型轻量检测生成优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。