通过动态替换不可靠视觉标记，显著减少视频大模型的幻觉问题。
MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models
- 引入轻量级视觉标记修补器，按需替换不可靠标记。
- 在Vript-HAL上使Qwen3-VL-4B-Instruct的F1提升50.60%。
- 不修改原模型，适合提升视频理解类任务的可靠性。
视频大型多模态模型在视频理解任务中取得显著进展，但仍易产生幻觉，即生成内容与输入视频不符。本文提出MultiToP，一种基于多模态上下文感知的视觉标记修补框架，通过在语言生成前优化不可靠视觉标记来缓解幻觉。该方法引入轻量级视觉标记修补器，预测标记级替换分布，并用动态全局补丁标记选择性替换不可靠标记。为有效训练修补器，提出信息引导的秩校准策略，利用骨干网络生成的答案条件帧级信息提示指导标记替换。结合真实答案监督与稀疏正则化，MultiToP实现局部视觉证据精炼，无需修改原始模型。大量实验表明，MultiToP在Vript-HAL上显著降低幻觉，对Qwen3-VL-4B-Instruct的F1分数提升50.60%；同时保持通用视频理解能力，在ActivityNet-QA上相对准确率提升18.58%。
原文摘要 · Abstract (English)
Video Large Multimodal Models have achieved remarkable progress in video understanding, yet they remain prone to hallucinations, where generated responses are not faithfully supported by the input video. In this paper, we propose MultiToP, a multimodal-context-aware visual token patching framework that mitigates hallucinations by refining unreliable visual tokens before language generation. MultiToP introduces a lightweight Visual Token Patcher to predict token-level replacement distributions and selectively substitute unreliable visual tokens with a dynamic global patch token. To train the patcher effectively, we further propose information-guided rank calibration, which uses answer-conditioned frame-level information cues derived from the backbone to guide token replacement. Combined with ground-truth answer supervision and sparsity regularization, MultiToP enables localized visual evidence refinement without modifying the original model. Extensive experiments demonstrate that MultiToP effectively reduces hallucinations on Vript-HAL with negligible inference overhead, improving the F1 scores of Qwen3-VL-4B-Instruct by 50.60% over the vanilla model. Meanwhile, MultiToP preserves general video understanding ability, yielding an 18.58% relative accuracy gain on ActivityNet-QA for Video-LLaVA-7B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。