让预训练模型摆脱概念捷径，提升公平性与鲁棒性
CURE: Controlled Unlearning for Robust Embeddings -- Mitigating Conceptual Shortcuts in Pre-Trained Language Models
- 通过内容提取器和反向网络分离无关概念信息
- 在IMDB上提升10点F1，Yelp上提升2点，开销极小
- 无需标注数据，适合追求公平性的NLP应用
预训练语言模型虽在多种任务中表现优异，但仍易受虚假概念关联影响，损害鲁棒性与公平性。本文提出CURE框架，通过专用内容提取器结合反向网络，系统性解耦并抑制概念捷径，同时保留关键语义信息。后续可控去偏模块利用对比学习精细调节残留概念线索的影响，可根据任务需求减弱有害偏差或保留有益关联。在IMDB和Yelp数据集上，采用三种预训练架构验证，CURE在IMDB上实现+10 F1绝对提升，在Yelp上提升+2，计算开销极低。该方法为对抗概念偏差提供了灵活、无监督的范式，推动更可靠、公平的语言理解系统发展。
原文摘要 · Abstract (English)
Pre-trained language models have achieved remarkable success across diverse applications but remain susceptible to spurious, concept-driven correlations that impair robustness and fairness. In this work, we introduce CURE, a novel and lightweight framework that systematically disentangles and suppresses conceptual shortcuts while preserving essential content information. Our method first extracts concept-irrelevant representations via a dedicated content extractor reinforced by a reversal network, ensuring minimal loss of task-relevant information. A subsequent controllable debiasing module employs contrastive learning to finely adjust the influence of residual conceptual cues, enabling the model to either diminish harmful biases or harness beneficial correlations as appropriate for the target task. Evaluated on the IMDB and Yelp datasets using three pre-trained architectures, CURE achieves an absolute improvement of +10 points in F1 score on IMDB and +2 points on Yelp, while introducing minimal computational overhead. Our approach establishes a flexible, unsupervised blueprint for combating conceptual biases, paving the way for more reliable and fair language understanding systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。