让视觉语言模型在推理时逐词纠错，大幅减少幻觉。
Token-Level Inference-Time Alignment for Vision-Language Models
- 用奖励模型在推理时生成细粒度反馈信号，无需重训练主模型。
- 在12个评测中表现提升，MMVet上准确率提高8.6%。
- 轻量高效，适合部署到现有视觉语言模型系统中。
视觉语言模型已成为现代多模态智能的核心，但其输出仍易产生与图像不符的幻觉。现有对齐方法通常依赖昂贵的微调或粗粒度的序列级推理策略，反馈延迟且精度不足。为此，我们提出TITA（Token-level Inference-Time Alignment），一个轻量级框架：冻结基础视觉语言模型，仅训练一个奖励模型以逼近其分布。推理时，通过奖励模型与目标模型之间的对数概率比提取隐式偏好信号，实现密集的自回归反馈。该方法可视为直接偏好优化（DPO）的推理时变体，在不重新训练主模型的前提下提供逐词修正信号。在LLaVA-1.5-7B和13B上的大量评估显示，12个基准测试中均有稳定提升，其中MMVet提升8.6%，POPE提升6.7%，表明理解能力更强、幻觉更少。在Qwen2.5-VL-7B和DeepSeek-VL2-27.5B上的实验也获得类似增益，尤其在减少幻觉和提升VQA准确率方面表现显著，且推理开销几乎可忽略。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have become essential backbones of modern multimodal intelligence, yet their outputs remain prone to hallucination-plausible text misaligned with visual inputs. Existing alignment approaches often rely on expensive fine-tuning with annotated preference data or sequence-level inference strategies that provide only coarse, delayed feedback. To overcome these limitations, we present TITA (Token-level Inference-Time Alignment), a lightweight framework that freezes the base VLM and instead trains a reward model to approximate its distribution. During inference, implicit preference signals are extracted as log-probability ratios between the reward model and the target VLM, yielding dense autoregressive feedback. This formulation can be viewed as an inference-time variant of Direct Preference Optimization (DPO), providing token-level corrective signals without retraining the backbone. Extensive evaluations on LLaVA-1.5-7B and 13B show consistent gains across 12 benchmarks, with improvements of 8.6% on MMVet and 6.7% on POPE, indicating stronger general understanding and reduced hallucinations. Additional experiments on Qwen2.5-VL-7B and DeepSeek-VL2-27.5B show comparable gains, especially in hallucination reduction and VQA accuracy, while incurring negligible inference overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。