提出无需微调的非线性去偏方法,按输入动态调整去偏策略。
BendVLM: Test-Time Debiasing of Vision-Language Embeddings
- 基于输入动态调整去偏方向,突破传统线性方法局限。
- 无需预先知道输入集合,适用于开放场景如图像生成与检索。
- 避免微调带来的灾难性遗忘,保持模型原有性能。
视觉-语言模型(VLM)嵌入常携带训练数据中的偏见,如对特定种族或性别群体的负面刻板印象。随着VLM被广泛应用于少样本分类、文本引导图像生成等任务,去偏至关重要。现有微调方法易导致灾难性遗忘,而无微调方法通常采用‘一刀切’的线性去偏策略,假设所有输入的虚假属性相关性可由单一方向解释。本文提出Bend-VLM,一种非线性、无需微调的VLM嵌入去偏方法,根据每个输入的独特性定制去偏操作,实现更灵活的去偏。此外,该方法无需在推理前预知输入集合,更适合在线、开放集任务,如信息检索与文本引导图像生成。
原文摘要 · Abstract (English)
Vision-language model (VLM) embeddings have been shown to encode biases present in their training data, such as societal biases that prescribe negative characteristics to members of various racial and gender identities. VLMs are being quickly adopted for a variety of tasks ranging from few-shot classification to text-guided image generation, making debiasing VLM embeddings crucial. Debiasing approaches that fine-tune the VLM often suffer from catastrophic forgetting. On the other hand, fine-tuning-free methods typically utilize a "one-size-fits-all" approach that assumes that correlation with the spurious attribute can be explained using a single linear direction across all possible inputs. In this work, we propose Bend-VLM, a nonlinear, fine-tuning-free approach for VLM embedding debiasing that tailors the debiasing operation to each unique input. This allows for a more flexible debiasing approach. Additionally, we do not require knowledge of the set of inputs a priori to inference time, making our method more appropriate for online, open-set tasks such as retrieval and text guided image generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。