用离散代码本让视觉模型适配特定领域，不改语言部分也能提效。
Decoupling Vision and Language: Codebook Anchored Visual Adaptation
- 用代码本锚定视觉表示，解耦视觉与语言模型的耦合关系。
- 在10个专业数据集上平均提升13.51%，保持语言能力不变。
- 适用于不同语言架构的模型，只需共享同一代码本即可复用。
大视觉语言模型（LVLMs）通过视觉编码器将图像转为表示以支持下游推理，但在医学影像诊断或细粒度分类等特定领域中，编码器表现常不足，导致表示错误在语言模型中传播并引发错误响应。现有方法通过投影器微调或参数高效更新来修改编码器与语言模型之间的连续特征接口，但依然存在组件耦合问题，一旦编码器变更即需重新对齐。我们提出CRAFT（Codebook RegulAted Fine-Tuning），一种轻量级方法，通过离散代码本对编码器进行微调，将视觉表示锚定至稳定标记空间，实现无需修改模型其他部分的领域适应。该解耦设计使适配后的编码器可无缝提升采用不同语言架构的LVLM性能，只要它们共享相同代码本。实证表明，CRAFT在包括VQARAD和PlantVillage在内的10个领域特定基准上平均提升13.51%，同时保留语言模型的语言能力，优于基于连续标记的方法。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-specific visual tasks such as medical image diagnosis or fine-grained classification, where representation errors can cascade through the language model, leading to incorrect responses. Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates, which still couples the two components and requires re-alignment whenever the encoder changes. We introduce CRAFT (Codebook RegulAted Fine-Tuning), a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space, achieving domain adaptation without modifying other parts of the model. This decoupled design allows the adapted encoder to seamlessly boost the performance of LVLMs with different language architectures, as long as they share the same codebook. Empirically, CRAFT achieves an average gain of 13.51% across 10 domain-specific benchmarks such as VQARAD and PlantVillage, while preserving the LLM's linguistic capabilities and outperforming peer methods that operate on continuous tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。