通过轻量激活调控，实现视觉语言模型的精准指令控制。
SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models
- 基于潜在嵌入学习动态调节跨模态激活。
- 仅需原模型0.14%参数即可实现推理时控制。
- 适合需要精准干预的多模态应用开发者。
本文提出SteerVLM，一种轻量级控制模块，用于引导视觉语言模型（VLMs）生成更符合目标指令的输出。该方法通过学习配对提示中目标与相反行为的潜在嵌入，动态调节语言模态与图像上下文间的激活连接，实现无需修改权重的细粒度推理时控制，同时保持非目标任务性能。模块参数量仅为原VLM的0.14%。通过维度级激活调制与分层自适应调控，无需预提取静态向量或人工设定干预点。此外，我们构建了VNIA（视觉叙事意图对齐）多模态数据集，专用于推进和评估VLM控制技术。实验表明，该方法在指令遵循与幻觉抑制基准上优于现有干预技术，为多模态模型控制提供了稳健的激活工程方案。
原文摘要 · Abstract (English)
This work introduces SteerVLM, a lightweight steering module designed to guide Vision-Language Models (VLMs) towards outputs that better adhere to desired instructions. Our approach learns from the latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context. This allows for fine-grained, inference-time control over complex output semantics without modifying model weights while preserving performance on off-target tasks. Our steering module requires learning parameters equal to 0.14% of the original VLM's size. Our steering module gains model control through dimension-wise activation modulation and adaptive steering across layers without requiring pre-extracted static vectors or manual tuning of intervention points. Furthermore, we introduce VNIA (Visual Narrative Intent Alignment), a multimodal dataset specifically created to facilitate the development and evaluation of VLM steering techniques. Our method outperforms existing intervention techniques on steering and hallucination mitigation benchmarks for VLMs and proposes a robust solution for multimodal model control through activation engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。