视觉语言模型用统一向量表示任务，跨模态通用且高效。
Vision-Language Models Create Cross-Modal Task Representations
- 用跨模态迁移发现任务向量，压缩后仍能准确触发生成。
- 仅用指令即可生成任务向量，无需示例，效果优于完整提示。
- 基础语言模型的任务向量可迁移到视觉语言模型中。
自回归视觉语言模型（VLMs）能统一处理多种任务，但其内部实现机制仍不清晰。我们发现，这类模型会将概念等价的输入对齐到一个共享任务向量中，该向量与模态（文本、图像）和格式（示例、指令）无关，可能简化模型处理流程。通过跨模态迁移测试——即在一种模态中提取的任务向量能否在另一模态中正确触发生成——我们在多个任务和模型架构上验证了这一现象。尽管任务向量高度压缩，其性能仍优于使用完整任务信息进行提示，这是跨模态场景下的独特优势。此外，我们证明任务向量可从基础语言模型迁移到微调后的视觉语言模型，并且仅需指令即可生成，无需示例。这些发现揭示了VLM如何内部处理任务信息，以及如何将不同模态映射到共同语义空间。
原文摘要 · Abstract (English)
Autoregressive vision-language models (VLMs) can handle many tasks within a single model, yet the representations that enable this capability remain opaque. We find that VLMs align conceptually equivalent inputs into a shared task vector, which is invariant to modality (text, image) and format (examples, instruction), and may simplify VLM processing. We measure this alignment via cross-modal transfer -- the ability of a task vector derived in one modality to trigger the correct generation in another -- on a range of tasks and model architectures. Although the task vector is highly compressed, we find that this single vector outperforms prompting the model with the full task information, unique to this cross-modal case. Furthermore, we show that task vectors can be transferred from a base language model to its fine-tuned vision-language counterpart, and that they can be derived solely from instructions without the need for examples. Taken together, our findings shed light on how VLMs internally process task information, and how they map different modalities into common semantic representations. Project page: https://vlm-cross-modal-reps.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。