arXiv:2608.22263cs.CVcs.AI2026-08

不微调模型,用三重对比解码实现精准个性化视觉理解。

Training-Free VLM Personalization via Calibrated Residual Decoding

论文配图:Training-Free VLM Personalization via Calibrated Residual Decoding
图 1 · 摘自论文原文
  • 通过正/反/空三种用户画像对比,量化个性化贡献
  • 在MMPB等数据集上显著提升身份敏感任务性能
  • 自适应不确定性校准,适合对隐私敏感的个性化场景

视觉语言模型可通过推理时直接输入用户画像、偏好或视觉参考实现免训练个性化,但直接提示难以确保模型真正利用这些信息。正向画像下的预测分布常混合个性化信号与模型固有先验,导致高置信度回答可能源于默认偏好而非真实用户意图。为此,我们提出一种免训练的校准残差解码框架:给定同一图像与问题,构建正向画像、反事实画像和空画像三种证据条件,以正向画像输出为锚点,通过三者得分差值显式估计个性化贡献,并引入基于归一化熵的不确定性校准,使个性化增强强度随残差信号可靠性自适应调整。在MMPB、YoLLaVA和MyVLM上的实验表明,该方法无需微调即可提升个性化多模态理解能力,尤其在身份敏感任务中表现稳定;额外分析显示,熵校准可有效缓解对比信号不确定时的解码波动。

原文摘要 · Abstract (English)

Vision-language models can be personalized in a training-free manner by directly providing user profiles, preferences, or visual references at inference time, without updating model parameters. However, direct personalized prompting does not guarantee that the model will reliably exploit such evidence. The predictive distribution under the positive user profile often mixes two sources: personalized signals genuinely supported by the current profile, and the model's generic visual or linguistic priors. As a result, from the positive-profile response alone, it is difficult to determine whether a high-confidence answer is supported by the user profile or merely reflects the model's default preference. To address this problem, we propose a training-free calibrated residual decoding framework. Given the same image and question, we construct three evidence conditions: a positive profile , a counterfactual profile , and an empty profile . Our method keeps the prediction under as the anchored base, and explicitly estimates the marginal contribution of personalization from score differences across the three conditions. We further introduce normalized-entropy-based uncertainty calibration, allowing the strength of personalized enhancement to adapt to the reliability of the residual signal. Experiments on MMPB, YoLLaVA, and MyVLM show that the proposed method improves personalized multimodal understanding without fine-tuning, with consistent gains on identity-sensitive visual personalization tasks. Additional analysis shows that entropy calibration stabilizes residual decoding when the contrastive personalization signal is uncertain.

个性化零样本多模态解码优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。