通过双流锚点校正,显著降低长图文生成中的对象幻觉。
Dual-Stream Cross-Anchor Correction Grounding Long-Form Captions and the Domain Limits of Object-Level Anchors
- 在微调中注入视觉锚点,分感知与认知双流协同对齐
- 生成长度达基线1.9倍,每对象提及准确率达88.19%
- 适用于图像内容真实场景,不适用于图表与幻觉图像
多模态大模型中的对象幻觉源于语言先验与语料共现偏差压倒视觉证据,导致对象提及无法对应图像。现有方法多在解码阶段干预,但在统一协议下仅对短文本有效;在细节丰富的语料上进行监督微调(SFT)虽可延长生成,但仍有超40%提及不存在对象。本文提出双流跨锚点校正(DSCC):在微调阶段将对象级视觉锚点注入语言模型——感知流通过双向对比目标,使中间层隐藏状态对齐冻结的文本锚点;认知流在每步生成时通过交叉注意力查询这些锚点;两阶段课程门控机制将其耦合,使证据检索成为生成的结构约束。在单一主干与评分协议下,实验覆盖长文本幻觉、对象存在性判别及跨域泛化。以相同语料和训练流程的原生SFT为对照,分离数据效应与架构优势。DSCC单独即进入长文本低幻觉区域:生成长度约为基线1.9倍,每对象提及精度达88.19%,为密度无关准则下的最高值。消融实验揭示协同效应:感知流单独使用会降低精度,但叠加认知流后反向提升。不宣称普适优势:三个域外基准显示可预测且可验证的域依赖性,协同效应受锚点语义域限制,在图表与幻觉图像上失效。
原文摘要 · Abstract (English)
Object hallucination in multimodal large language models arises when language priors and corpus co-occurrence bias outweigh the visual evidence, with nothing tying an object mention to the image. Most remedies intervene at decoding time, yet under a unified protocol their benefit is confined to short captions; supervised fine-tuning (SFT) on a detail-rich corpus lengthens captions, but over forty percent still name absent objects. This paper proposes Dual-Stream Cross-Anchor Correction (DSCC). Unlike work that post-processes decoding, DSCC injects object-level visual anchors into the language model itself during fine-tuning: a perception stream aligns object-level hidden states at an intermediate layer to frozen text anchors by a bidirectional contrastive objective; a cognition stream lets deeper layers query those anchors by cross-attention at every generation step; and a two-stage curriculum gate couples them, making evidence retrieval a structural constraint on generation. Under one backbone and one scoring protocol, experiments span long-caption hallucination, object-existence discrimination and cross-domain generalisation, with vanilla SFT on the same corpus and schedule as a length- and density-matched control separating the data effect from the architectural gain. DSCC alone reaches the long-caption, low-hallucination region: captions roughly 1.9 times the baseline length at 88.19% precision per object mention, the highest under a density-independent criterion. Ablations expose a synergy: the perception stream alone degrades precision yet reverses sign when stacked on the cognition stream. No universal superiority is claimed: three out-of-domain benchmarks yield a predictable, falsifiable domain-conditionality, the synergy being bound to the anchors' semantic domain and breaking on charts and illusions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。