arXiv:2507.19882cs.AI2025-07被引 3

用扩散模型生成反事实样本,让提示词更符合数据因果关系。

Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generation

  • 通过扩散过程迭代采样梯度,生成满足最小充分性的反事实样本。
  • 在图像分类等任务中,对未见类别表现显著优于现有方法。
  • 适合需要强泛化能力的因果推理场景,尤其适用于少样本学习。

提示学习因其高效性受到关注,但现有方法因理论基础不足,难以获得因果不变的提示,无法有效捕捉跨类别泛化的鲁棒特征。为此,我们提出基于扩散的反事实提示学习框架DiCap,利用扩散过程从因果模型的边缘与条件分布中迭代采样梯度,生成满足最小充分性准则的反事实样本。该方法在严格理论推导基础上,保证了反事实结果的可识别性,并对估计误差施加严格边界。进一步采用对比学习框架,利用生成的反事实样本,精确提取与数据因果特征对齐的提示。大量实验表明,该方法在图像分类、图文检索和视觉问答等任务上表现优异,尤其在未见类别上优势明显。

原文摘要 · Abstract (English)

Prompt learning has garnered attention for its efficiency over traditional model training and fine-tuning. However, existing methods, constrained by inadequate theoretical foundations, encounter difficulties in achieving causally invariant prompts, ultimately falling short of capturing robust features that generalize effectively across categories. To address these challenges, we introduce the $\textit{\textbf{DiCap}}$ model, a theoretically grounded $\textbf{Di}$ffusion-based $\textbf{C}$ounterf$\textbf{a}$ctual $\textbf{p}$rompt learning framework, which leverages a diffusion process to iteratively sample gradients from the marginal and conditional distributions of the causal model, guiding the generation of counterfactuals that satisfy the minimal sufficiency criterion. Grounded in rigorous theoretical derivations, this approach guarantees the identifiability of counterfactual outcomes while imposing strict bounds on estimation errors. We further employ a contrastive learning framework that leverages the generated counterfactuals, thereby enabling the refined extraction of prompts that are precisely aligned with the causal features of the data. Extensive experimental results demonstrate that our method performs excellently across tasks such as image classification, image-text retrieval, and visual question answering, with particularly strong advantages in unseen categories.

提示学习因果推理扩散模型反事实生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。