用分布差异正则化，让大模型少依赖表面特征，提升泛化能力。
DBR: Divergence-Based Regularization for Debiasing Natural Language Understanding Models
- 通过对比原句与屏蔽关键词后的输出分布，识别并抑制模型对表面特征的依赖。
- 在三个NLU任务上，域外性能显著提升，域内准确率几乎不变。
- 适合关注模型泛化性、避免过拟合表面模式的研究者使用。
预训练语言模型（PLMs）在多项自然语言处理任务中取得了显著成果。然而，近期研究发现这些模型常依赖表面特征和捷径，而非真正理解语言，尤其在自然语言理解（NLU）任务中表现明显，导致其难以泛化到域外数据。本文提出基于分布差异的正则化方法（DBR），通过测量原始样本与屏蔽快捷特征词后样本的输出分布差异，防止模型预测过度受表面特征影响。我们在三个NLU任务上评估该方法，结果表明其在保持域内精度基本不变的前提下，显著提升了域外性能。实验验证了减少对捷径和表面特征的依赖，可有效增强大型预训练语言模型的泛化能力。
原文摘要 · Abstract (English)
Pre-trained language models (PLMs) have achieved impressive results on various natural language processing tasks. However, recent research has revealed that these models often rely on superficial features and shortcuts instead of developing a genuine understanding of language, especially for natural language understanding (NLU) tasks. Consequently, the models struggle to generalize to out-of-domain data. In this work, we propose Divergence Based Regularization (DBR) to mitigate this shortcut learning behavior. Our method measures the divergence between the output distributions for original examples and examples where shortcut tokens have been masked. This process prevents the model's predictions from being overly influenced by shortcut features or biases. We evaluate our model on three NLU tasks and find that it improves out-of-domain performance with little loss of in-domain accuracy. Our results demonstrate that reducing the reliance on shortcuts and superficial features can enhance the generalization ability of large pre-trained language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。