通过新预训练任务提升模型对否定句的理解能力。
Making Language Models Robust Against Negation
- 设计新任务NSPP，让模型学习句子极性变化。
- 在九个否定相关数据集上表现更优，最高提升9.1%。
- 适合需要准确理解否定语义的问答与推理场景。
否定一直是语言模型面临的长期挑战。以往研究显示，模型在多项自然语言理解任务中难以处理否定。本文提出一种自监督方法，通过引入新的任务——下一句极性预测（NSPP）及其变体，对BERT和RoBERTa进行进一步预训练。实验表明，经过该方法微调的模型在九个与否定相关的基准测试中均优于原始版本。尤其在需基于否定进行推理的大型问答数据集CondaQA上，性能提升达1.8%至9.1%。
原文摘要 · Abstract (English)
Negation has been a long-standing challenge for language models. Previous studies have shown that they struggle with negation in many natural language understanding tasks. In this work, we propose a self-supervised method to make language models more robust against negation. We introduce a novel task, Next Sentence Polarity Prediction (NSPP), and a variation of the Next Sentence Prediction (NSP) task. We show that BERT and RoBERTa further pre-trained on our tasks outperform the off-the-shelf versions on nine negation-related benchmarks. Most notably, our pre-training tasks yield between 1.8% and 9.1% improvement on CondaQA, a large question-answering corpus requiring reasoning over negation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。