通过对抗性分词优化,显著降低多模态大模型幻觉率。
TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs
- 将偏好优化重构为最小-最大问题,动态扰动无关视觉的词元。
- 仅用4800条数据,幻觉率从26.4%降至13.2%,认知评分降为0.4。
- 适合关注幻觉抑制与高效训练的多模态模型研究者。
多模态大语言模型易产生幻觉,生成看似合理但缺乏视觉依据的内容,部分原因在于静态偏好监督下直接偏好优化(DPO)过度拟合表面语言线索。本文提出TARS,一种基于词元自适应的偏好策略,将DPO重构为严谨的最小-最大优化问题。内层最大化通过选择性扰动视觉无关词元,诱导最坏情况下的分布偏移;外层最小化则强制模型对齐因果视觉信号而非表面模式。新颖的谱对齐损失利用快速傅里叶变换(FFT)在频域正则化隐藏表示,保留全局语义结构且无需严格词元对应。我们在多个幻觉基准上评估TARS。仅使用4.8k偏好样本(无专家标注),幻觉率由26.4%降至13.2%,认知评分从2.5降至0.4,大幅优于标准DPO。值得注意的是,TARS优于使用28.8k样本的五倍数据增强方法(幻觉率16.0% vs. 13.2%),表明通过对抗性词元扰动重塑优化空间比单纯扩充数据更有效。TARS进一步缩小了与GPT-4o在关键指标上的差距。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) are prone to hallucinations, generating plausible but visually ungrounded outputs, partly because direct preference optimization (DPO) overfits to superficial linguistic cues under static preference supervision. We propose TARS, a token-adaptive preference strategy that reformulates DPO as a principled min-max optimization problem. The inner maximization selectively perturbs visual-agnostic tokens to induce worst-case distributional shifts, while the outer minimization enforces alignment with causal visual signals rather than surface-level patterns. A novel spectral alignment loss further regularizes hidden representations in the frequency domain via the Fast Fourier Transform (FFT), preserving global semantic structure without rigid token-level correspondence. We evaluate TARS across multiple hallucination benchmarks. Using only 4.8k preference samples without expert feedback, TARS reduces hallucination rates from 26.4\% to 13.2\% and cognition scores from 2.5 to 0.4, outperforming standard DPO by a large margin. Notably, TARS surpasses $5\times$ LLM-based data augmentation trained on 28.8k samples (Hal-Rate: 16.0\% vs.\ 13.2\%), demonstrating that reshaping the optimization landscape via adversarial token perturbation is fundamentally more effective than scaling training data. TARS further narrows the gap with GPT-4o on key metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。