arXiv:2608.21486cs.CV2026-08中稿 · ECCV

让人脸识别模型自己说出判断依据,无需标注数据。

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

论文配图:EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment
图 1 · 摘自论文原文
  • 用轻量适配器对齐视觉语言模型与冻结的人脸识别空间,实现语义解释。
  • 仅用图像训练,通过100个可检测属性构建模型可读语义签名,提升辨识能力。
  • 支持无标签、端到端的属性审计,适合安全评估与公平性分析。

深度人脸识别(FR)模型已接近饱和准确率,但其决策过程仍不透明:从业者无法得知相似度评分依赖哪些语义属性。EXPL-FR在FR模型自身的嵌入空间中回答这一问题。一个轻量级适配器将视觉语言模型(VLM)的图像编码器与冻结的FR空间对齐,仅使用人脸图像训练,从未接触文本。由于VLM编码器共享同一空间,相同的适配器也可用于文本编码器,将22类共978个属性提示转化为FR空间锚点,且无需额外开销。我们不假设该迁移有效:通过人脸验证协议进行测量,并通过仅改变适配器的消融实验隔离其贡献。并非所有概念都能保留,因为FR模型通过丢弃需跨身份验证的因子来获得不变性。一种无标签可检测性度量比较每个概念在FR空间与VLM空间中的可分性,前100个最可检测的概念构成模型的可读语义签名,其分离身份的能力优于完整词汇表。我们测试了四种FR主干网络和两种VLM编码器,EXPL-FR无需架构访问,支持个体级、单图及差异解释。我们在三种监督设置下进行属性级审计:人类标注(现有实践)、VLM伪标签和完全基于提示的审计,对比真实验证行为。无标签情况下,提示驱动审计能按种族层面的RFW误差排名四类模型,并按真实验证成本排序受控属性变化。

原文摘要 · Abstract (English)

Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only the adapter isolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-free detectability measure compares each concept's separability in FR space against the VLM space, and the 100 most detectable form the model's readable semantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fully prompt-driven audit, against real verification behavior. With no labels, the prompt-driven audit ranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.

人脸识别可解释性视觉语言模型无监督审计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。