通过稀疏编码空间精准调节模型偏见，提升视觉语言模型公平性
SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models
- 在稀疏自编码器潜空间中分离并调控与偏见相关的神经元
- 在4个数据集上实现显著公平性提升，零样本分类准确率提高5.2%以上
- 无需微调，适用于多种CLIP模型，适合研究与应用中的去偏需求
连接视觉与语言的模型（如CLIP）是多模态AI的核心，但其大规模、未经筛选的训练数据引入了严重的社会偏见和虚假关联。现有后处理去偏方法通常在密集的CLIP嵌入空间中操作，导致偏见与任务相关特征高度纠缠，难以在不损害语义保真度的前提下去除偏见。本文提出稀疏嵌入调制（SEM），一种后处理、零样本的去偏框架，基于稀疏自编码器（SAE）潜空间工作。通过将CLIP文本嵌入分解为解耦特征，SEM识别并调制与偏见相关的神经元，同时保留与查询相关的特征，实现更精确的非线性干预。在四个基准数据集及两种CLIP主干网络上，SEM在检索和零样本分类任务中均取得显著的公平性提升。结果表明，稀疏潜表示为视觉语言模型的后处理去偏提供了有效基础。
原文摘要 · Abstract (English)
Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods often operate directly in the dense CLIP embedding space, where bias and task-relevant information are highly entangled. This entanglement limits their ability to remove bias without degrading semantic fidelity. In this work, we propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder (SAE) latent space. By decomposing CLIP text embeddings into disentangled features, SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones. This enables more precise, non-linear interventions. Across four benchmark datasets and two CLIP backbones, SEM achieves substantial fairness gains in retrieval and zero-shot classification. Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。