提出新指标评估CLIP对语义改写敏感度，发现性别相关查询表现差异。
PRSM: A Measure to Evaluate CLIP's Robustness Against Paraphrases
- 设计PRSM指标量化CLIP对改写查询的响应稳定性
- 在社会反事实数据集上发现男性/女性关联词查询稳定性不同
- 揭示多模态模型在敏感场景下可能放大数据偏见
对比语言-图像预训练（CLIP）是广泛使用的多模态模型，通过大规模训练对齐文本与图像表征。尽管其在零样本和少样本任务中表现优异，但对语言变化（尤其是改写）的鲁棒性仍缺乏研究。改写鲁棒性对可靠部署至关重要，尤其在涉及社会敏感性的场景中，不一致的表征可能加剧人口偏差。本文提出一种新的衡量指标——改写排序稳定性度量（Paraphrase Ranking Stability Metric, PRSM），用于量化CLIP对改写查询的敏感性。基于专为揭示社会与人口偏差设计的社交反事实数据集（Social Counterfactuals），我们实证评估了CLIP在改写变异下的稳定性，分析了改写鲁棒性与性别之间的交互关系，并讨论了对公平性和多模态系统公平部署的影响。结果表明，不同改写策略下的鲁棒性存在差异，且在与男性/女性相关的查询中观察到细微但持续的差异。
原文摘要 · Abstract (English)
Contrastive Language-Image Pre-training (CLIP) is a widely used multimodal model that aligns text and image representations through large-scale training. While it performs strongly on zero-shot and few-shot tasks, its robustness to linguistic variation, particularly paraphrasing, remains underexplored. Paraphrase robustness is essential for reliable deployment, especially in socially sensitive contexts where inconsistent representations can amplify demographic biases. In this paper, we introduce the Paraphrase Ranking Stability Metric (PRSM), a novel measure for quantifying CLIP's sensitivity to paraphrased queries. Using the Social Counterfactuals dataset, a benchmark designed to reveal social and demographic biases, we empirically assess CLIP's stability under paraphrastic variation, examine the interaction between paraphrase robustness and gender, and discuss implications for fairness and equitable deployment of multimodal systems. Our analysis reveals that robustness varies across paraphrasing strategies, with subtle yet consistent differences observed between male- and female-associated queries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。