当屏幕文字与画面冲突时,视觉语言模型会错误依赖文字,本文提出新评测和解决方案。
When Text Hijacks Vision: Benchmarking and Mitigating Text Overlay-Induced Hallucination in Vision Language Models

- 构建首个图文干扰幻觉评测集,量化文本与视觉矛盾程度
- 提出混合专家模型,动态分配专家处理时间、动作、物体、空间四类任务
- 在6057个样本上验证有效,显著减少幻觉且不降低正常视频表现
视觉语言模型在多模态视频理解任务中表现优异,但本文发现一个关键问题:当屏幕上文字与真实画面矛盾时,现有模型会系统性地产生幻觉,优先采纳文字语义而非视觉内容。作者将此现象定义为文本叠加诱导幻觉(TOIH)。为此,提出首个全面的评测基准VisualTextTrap,基于公开数据集通过自动化与人工结合的方式构建,包含6,057个样本,覆盖88个细粒度属性,分四个维度评估,幻觉强度用五级量表(L1-L5)量化。同时提出新型视觉-文本解耦框架VTHM-MoE,采用双编码器结构,预训练四个针对时间、动作、物体、空间的专用专家模块,通过自适应令牌路由策略实现动态专家分配,在多种视频问答任务中显著优于当前最优方法,有效抑制幻觉且保持原性能。
原文摘要 · Abstract (English)
Recent advances in Vision-Language Models (VLMs) have substantially enhanced their ability across multimodal video understanding benchmarks spanning temporal, action, object, and spatial understanding. However, we identify a critical yet overlooked issue: when embedded on-screen text contradicts the visual scene, existing VLMs systematically hallucinate, prioritizing overlay textual semantics over the actual visual content. We define this phenomenon as Text Overlay-Induced Hallucination (TOIH). In this work, we propose VisualTextTrap, the first comprehensive benchmark, including large-scale human-validated samples with specifically designed evaluation metrics. In particular, we construct VisualTextTrap from widely-used public datasets using a scalable hybrid pipeline of VLMs assisted text generation and rigorous manual verification. The benchmark features 6,057 samples annotated across 88 fine-grained attributes within four dimensions, with hallucination intensity quantified on a five-level scale (L1--L5) that reflects the semantic contradiction between overlay text and visual reality. Moreover, we propose Visual Text Hallucination Mitigation Mixture-of-Experts (VTHM-MoE), a novel Vision-Text Disentanglement framework that employs a dual-encoder architecture. Concretely, four dimension-specialized expert modules spanning Temporal, Action, Object, and Spatial reasoning are first pre-trained to identify and leverage cross-modal discrepancies between textual semantics and actual video content. We develop an Adaptive Token Routing Strategy to enable dynamic expert allocation, conferring robust resistance to TOIH while preserving performance on uncontaminated videos. Extensive experiments conducted on our VisualTextTrap benchmark verify the effectiveness of VTHM-MoE, outperforming state-of-the-art counterparts with diverse video question answering tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。