不依赖分组标注，用极少量参数提升模型对少数群体的鲁棒性。
Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness
- 通过投影图像特征到文本编码器类代理的零空间，识别偏差分类器中的少数样本。
- 在无需训练分组标签的情况下，平均最差组准确率超越现有方法，仅需0.01%可调参数。
- 理论证明该方法能最优降低均衡组误差，适合资源受限场景下的高效微调。
尽管图像-文本基础模型在各类下游任务中表现优异，但仍面临输入与标签间虚假相关性的问题。为此，我们提出一种简单三步法——项目-探测-聚合（PPA），实现无需依赖分组标注的基础模型参数高效微调。基于失败驱动去偏策略，本方法改进了两个关键组件：少数样本识别与鲁棒训练算法。首先，通过将图像特征投影至文本编码器类代理的零空间，训练偏差分类器；其次，利用该分类器推断组标签，并结合先验校正探测组目标；最后，聚合每类的组权重生成去偏分类器。理论分析表明，PPA能增强少数群体识别能力，且在最小化均衡组误差上为贝叶斯最优，有效缓解虚假相关性。大量实验验证其有效性：在不训练组标签的前提下，平均最差组准确率超越当前最优方法，且可调参数不足0.01%。
原文摘要 · Abstract (English)
While image-text foundation models have succeeded across diverse downstream tasks, they still face challenges in the presence of spurious correlations between the input and label. To address this issue, we propose a simple three-step approach,Project-Probe-Aggregate (PPA), that enables parameter-efficient fine-tuning for foundation models without relying on group annotations. Building upon the failure-based debiasing scheme, our method, PPA, improves its two key components: minority samples identification and the robust training algorithm. Specifically, we first train biased classifiers by projecting image features onto the nullspace of class proxies from text encoders. Next, we infer group labels using the biased classifier and probe group targets with prior correction. Finally, we aggregate group weights of each class to produce the debiased classifier. Our theoretical analysis shows that our PPA enhances minority group identification and is Bayes optimal for minimizing the balanced group error, mitigating spurious correlations. Extensive experimental results confirm the effectiveness of our PPA: it outperforms the state-of-the-art by an average worst-group accuracy while requiring less than 0.01% tunable parameters without training group labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。