arXiv:2605.31429cs.CV2026-05

通过共享浅层计算,用注册令牌实现高效幻觉抑制。

YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models

论文配图:YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models
图 1 · 摘自论文原文
  • 在中间层分支构建退化模型,共享浅层计算避免重复推理。
  • 使用注册令牌保留全局语义,降低局部细节以提供精准对比信号。
  • 无需训练,显著降低延迟,适用于多类视觉语言模型。

对比解码(CD)通过对比标准模型与视觉退化模型的输出分布来缓解大型视觉语言模型(LVLM)中的幻觉问题。然而,现有无训练的CD方法存在退化分支不理想的问题:完全丢弃视觉标记过于极端,会引发语言幻觉;而图像扰动则对视觉证据控制粗糙,且因需两次完整前向传播导致高推理延迟。为此,我们提出YARD——一种无训练的Y架构注册解码框架。受观察启发:可靠的文本到视觉对齐主要出现在中间解码层,YARD通过共享浅层计算,并在该关键阶段分支构造退化分支。退化分支将像素级视觉标记替换为注册标记,保留全局图像语义但缺乏细粒度局部证据。这种兼顾图像感知且局部弱对齐的设计,提供了忠实的对比信号,同时避免了模态失配。此外,Y架构严格避免二次前向传播,显著降低延迟。在生成式与判别式幻觉基准上的大量实验表明,YARD在多个LVLM上均实现了领先的幻觉抑制效果,并大幅减少推理延迟。

原文摘要 · Abstract (English)

Contrastive decoding (CD) seeks to mitigate hallucinations in Large Vision-Language Models (LVLMs) by contrasting the output distributions of a standard model and a visually degraded model. However, existing training-free CD methods suffer from sub-optimal degraded branches: completely dropping visual tokens is too extreme and induces language hallucinations, while corrupting input images offers coarse control over visual evidence and suffers from high inference latency due to requiring two full forward passes. To address these dilemmas, we propose YARD, a training-free Y-Architecture Register Decoding framework. Motivated by the observation that reliable text-to-vision grounding predominantly emerges in the middle decoder layers, YARD constructs the degraded branch internally by sharing shallow-layer computations and branching exactly at this critical stage. For the degraded branch, YARD replaces patch-level visual tokens with register tokens, which preserve global image semantics but lack fine-grained local evidence. This image-aware yet locally under-grounded design provides a faithful contrastive signal without extreme modality mismatch, while the Y-architecture strictly avoids a costly second forward pass. Extensive experiments on generative and discriminative hallucination benchmarks demonstrate that YARD consistently achieves state-of-the-art hallucination mitigation across multiple LVLMs, alongside a significant reduction in inference latency.

幻觉抑制视觉语言模型推理加速对比解码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。