arXiv:2511.22897cs.CV2025-11

用分布代替点,让多模态提示更鲁棒

From Points to Clouds: Learning Robust Semantic Distributions for Multi-modal Prompts

  • 将提示学习从单点优化转为动态去噪,构建语义分布
  • 在11个数据集上表现最优,新类别泛化提升1.4%
  • 适合追求跨模态对齐鲁棒性的研究者

多模态提示学习(MPL)是适配大规模视觉语言模型(VLMs)的关键技术。现有方法受限于单一静态点表示,易过拟合基础类别,对新类别或模糊类别泛化能力差。本文挑战这一点表示范式,提出学习语义云(嵌入空间中的分布)以实现鲁棒泛化。为此,我们提出Points-to-Clouds(P2C)框架,受扩散模型启发,将提示学习重构为动态去噪任务。核心是双去噪机制:动态提示去噪(DPD)通过渐进噪声扰动文本提示,学习更平滑的语义空间;辅助视觉-语言映射器去噪损失将映射器重构成去噪自编码器,使其从噪声文本输入中重建干净视觉提示,确保跨模态对齐鲁棒性。在11个数据集上的实验表明,P2C持续优于强基线,在基础到新类别的泛化基准上达到79.7%的调和均值,相对基线提升1.4%。代码与模型已开源。

原文摘要 · Abstract (English)

Multimodal Prompt Learning (MPL) has emerged as a pivotal technique for adapting large-scale Visual Language Models (VLMs). However, current MPL methods are fundamentally limited by their optimization of a single, static point representation. This paradigm is inherently brittle, leads to overfitting on base classes, and generalizes poorly to novel or ambiguous categories. We challenge this point paradigm, proposing that robust generalization requires learning a semantic cloud (i.e., a distribution over the embedding space). To achieve this, we introduce Points-to-Clouds (P2C), a novel framework inspired by diffusion models that reframes prompt learning as a dynamic denoising task. At the core of P2C is a dual denoising mechanism: a Dynamic Prompt Denoising (DPD) mechanism perturbs text prompts with sophisticated, annealed noise to learn a smoother semantic landscape, while an auxiliary V-L Mapper denoising loss re-tasks the mapper as a denoising autoencoder. This forces the mapper to reconstruct clean visual prompts from noisy text inputs, ensuring robust cross-modal alignment. Extensive experiments across 11 datasets demonstrate that P2C consistently outperforms strong baselines. On the base-to-novel generalization benchmark, our method achieves a Harmonic Mean of 79.7%, representing a relative improvement of 1.4% over the baseline. The code and models are available at https://vranlee.github.io/P2C/.

多模态提示扩散模型语义分布

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。