arXiv:2608.11474cs.CV2026-08中稿 · ECCV

提出一种无需训练的去幻觉方法,提升视觉语言模型生成准确性

Test-Time Hallucination Control in Large Vision-Language Models

论文配图:Test-Time Hallucination Control in Large Vision-Language Models
图 1 · 摘自论文原文
  • 引入零样本多模态分类器作为令牌验证器,融合图像信息生成辅助置信度
  • 在多个模型和基准上显著降低幻觉率,准确率平均提升5.3%以上
  • 仅需单次解码,不修改模型内部状态,适合实际部署

大型视觉语言模型(LVLM)中的物体幻觉问题——即生成与输入图像无关的虚假内容——仍是其在真实场景中应用的关键障碍。现有缓解策略分为基于训练和无训练两类。前者虽效果好但成本高,后者更高效但常需多次解码或破坏预训练知识。本文提出测试时幻觉控制(TTH),一种新型无训练方法,同时解决上述缺陷。TTH引入一个零样本多模态分类器(MMC)作为令牌验证器,生成基于输入图像的辅助逻辑值。这些逻辑值在选定候选对象的词元层级与原模型输出融合,并通过熵加权机制实现鲁棒预测。跨多个LVLM系列和多样基准的大量实验表明,TTH在准确性和鲁棒性上持续提升,彰显其通用性与实用性。代码已开源。

原文摘要 · Abstract (English)

Object Hallucination in large vision-language models (LVLMs), where models generate non-factual content about input images, remains a critical barrier to their reliability in real-world applications. Existing mitigation strategies can be categorized into training-based and training-free methods. Training-based methods often achieve strong performance but are costly, requiring extensive computational resources, large-scale data, and time-consuming fine-tuning. Training-free approaches are particularly appealing due to their efficiency. However, existing training-free methods either require multiple decoding rounds, which adds computational overhead, or modify internal states in a model-specific way that risks degrading pretrained knowledge. We propose Test-Time Hallucination Mitigation (TTH) method, a novel training-free method that addresses both limitations. TTH introduces a token-validator module, implemented as a zero-shot Multi-Modal Classifier (MMC), to generate auxiliary logits grounded in the input image. These logits are fused with the original LVLM outputs at the token level for object tokens selected from a candidate pool. An entropy-based weighting scheme is then applied to enable robust and accurate predictions. Extensive experiments across multiple LVLM families and diverse benchmarks demonstrate that TTH consistently improves accuracy and robustness, underscoring its generalizability and practical effectiveness. Code is released at https://github.com/Mehran-TAM/TTH

视觉语言模型幻觉控制无训练方法多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。