融合双模型检测视觉语言模型幻觉，提升定位精度与置信度校准。
SpanCalib-VLM: Calibrated Hallucination Span Detection in Vision-Language Models
- 用交叉注意力融合文本与图像编码器，构建多模态序列标注器。
- 联合生成式模型与判别式模型，实现0.41的校准相关性与70.7%检测准确率。
- 适合关注幻觉检测可靠性的研究人员和应用开发者。
大型视觉语言模型（LVLMs）中的幻觉检测需兼具精确的片段定位与校准的置信度评分。微调的生成式VLM在识别幻觉文本片段方面表现优异，但存在过度自信和高推理延迟问题；判别式序列标记器虽具备确定性速度和更优校准性，却召回率偏低。本文提出SpanCalib-VLM，一种用于SHROOM-Visions共享任务的混合双系统，结合了由XLM-RoBERTa-Large与SigLIP视觉编码器通过交叉注意力融合的多模态序列标记器，以及微调的生成式VLM（Qwen3.5-4B-SHROOM-SFT）。通过联合校准融合策略，生成模型的候选片段由序列标记器提供的校准概率重新评分。在SHROOM-Visions英文评估集上，该集成模型取得0.41的皮尔逊校准相关性、0.39的整体交并比、0.91的干净响应交并比及70.7%的整体检测准确率。模型权重与代码已公开。
原文摘要 · Abstract (English)
Detecting hallucinations in Large Vision-Language Models (LVLMs) requires both accurate span localization and well-calibrated confidence scores. Fine-tuned generative VLMs excel at identifying hallucinated text spans but suffer from overconfidence and high inference latency. Discriminative sequence taggers offer deterministic speed and superior calibration but exhibit conservative span recall. We present SpanCalib-VLM, a hybrid dual-system for the SHROOM-Visions Shared Task that combines a multimodal sequence tagger, consisting of XLM-RoBERTa-Large fused with a SigLIP vision encoder via cross-attention, with our fine-tuned generative VLM (Qwen3.5-4B-SHROOM-SFT). Through a Union-Calibrated Fusion strategy, candidate spans from the generative model are re-scored with calibrated probabilities from the sequence tagger. On the SHROOM-Visions English evaluation split, our ensemble achieves a Pearson calibration correlation of 0.41 and an overall IoU of 0.39, with a clean-response IoU of 0.91} and overall detection accuracy of 70.7%. We make our model weights and code publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。