用一次优化生成通用引导向量,有效减少视觉语言模型幻觉。
One-shot Optimized Steering Vector for Hallucination Mitigation for VLMs
- 基于语义对齐选择样本,单次优化生成通用引导向量。
- 在多个基准上显著降低幻觉率,推理开销几乎为零。
- 适合追求高效安全的视觉语言模型部署场景。
视觉语言模型(VLMs)在多模态任务中表现强劲,但仍存在幻觉和安全问题,且规模扩大后仍持续存在。引导(steering)是一种轻量级提升性能的方法,但输入依赖与非依赖方案难以兼顾效率与效果。本文观察到:当任务具有语义一致意图时,引导向量可在不同输入间泛化。据此提出OSGA(One-shot Steering with Generative Anchor),一种无需输入依赖的框架,仅需一次优化即可生成通用引导向量。OSGA通过方差策略选取信息丰富的样本,利用对比学习与生成锚正则化训练单一引导向量。该向量可在推理时固定层应用,无需修改模型参数。跨多个基准实验表明,单个OSGA优化向量能持续提升幻觉缓解与安全性,开销极低,验证了单次引导作为可靠VLM实用可扩展解决方案的潜力。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) achieve strong performance on multimodal tasks but still suffer from hallucination and safety-related failures that persist even at scale. Steering offers a lightweight technique to improve model performance. However, steering, whether input-dependent or input-independent, achieves a meaningful trade-off between efficiency and effectiveness. In this work, we observe that steering vectors can generalize across inputs when tasks share aligned semantic intent. Based on this insight, we propose \textbf{OSGA} (\textbf{O}ne-shot \textbf{S}teering with \textbf{G}enerative \textbf{A}nchor), an input-independent framework that improves model performance with a single optimization instance. OSGA first selects an informative sample via a variance-based data selection strategy and learns a single steering vector with a contrastive objective with generative anchor regularization. The resulting vector can be universally applied at a certain layer during inference time without modifying model parameters. Experiments across multiple benchmarks show that a single OSGA-optimized steering vector consistently improves hallucination mitigation and safety enhancement with negligible overhead, highlighting one-shot steering as a practical and scalable solution for reliable VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。