让图文模型更专注关键信息，减少幻觉
Differential Multimodal Transformers
- 将差分注意力机制引入图文模型，提升关键信息筛选能力
- 在PaliGemma 3B上微调后，问答准确率显著提升
- 适合希望降低幻觉、优化多模态推理的开发者
小型语言模型因高效和日益增强的能力而受到广泛关注。然而，引入视觉等额外模态会因噪声增加而加剧上下文窗口有限的问题。近期研究指出，Transformer注意力机制常过度关注无关上下文。本文将原本用于纯文本模型的差分注意力机制扩展至图文模型PaliGemma。通过使用LoRA对PaliGemma 3B模型进行微调，并测试不同参数设置与配置，验证其在缓解噪声信息检索和减少幻觉方面的有效性。结果表明，差分注意力可成功集成到现有模型的微调流程中，显著提升噪声环境下信息检索与问答表现。
原文摘要 · Abstract (English)
Small language models have gained significant popularity due to their efficiency and growing capabilities. However, incorporating additional modalities, such as vision, can exacerbate the challenge of limited context windows by introducing noise. Recent studies have highlighted that Transformer attention mechanisms often disproportionately focus on irrelevant contexts. In this work, we extend the Differential Attention mechanism, originally designed for text-only models, to the text-vision model PaliGemma. Our aim is to evaluate its ability to mitigate noisy information retrieval and reduce hallucinations. To this end, we fine-tuned the PaliGemma 3B model using LoRA, incorporating Differential Attention, and experimented with various parameter settings and configurations. We demonstrate that Differential Attention can be adapted and integrated into the fine-tuning of existing models to enhance noisy information retrieval and question-answering capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。