用结构感知与大模型生成数据,提升模型抗干扰和泛化能力
SALAD: Improving Robustness and Generalization through Contrastive Learning with Structure-Aware and LLM-Driven Augmented Data
- 基于标签生成结构相似正样本,利用大模型构造反事实负样本
- 在情感分类等任务中显著提升分布外数据表现
- 适合需要强鲁棒性与跨领域泛化的NLP应用
在自然语言处理任务中,微调预训练语言模型常引发虚假相关性问题,尤其在分布外数据上表现不佳。为此,我们提出SALAD(Structure Aware and LLM-driven Augmented Data)方法,通过生成结构感知的正样本和大语言模型驱动的反事实负样本,结合对比学习提升模型鲁棒性与泛化能力。该方法采用基于标签的策略生成结构一致的正样本,并利用大语言模型生成具有多样句式结构的反事实负样本。对比学习使模型更关注关键成分间的结构关系,减少对虚假线索的依赖。我们在情感分类、性别歧视检测和自然语言推理三个任务上验证了该方法,结果表明SALAD不仅提升了不同环境下的性能,还在分布外数据和跨领域场景中展现出更强的泛化能力。
原文摘要 · Abstract (English)
In various natural language processing (NLP) tasks, fine-tuning Pre-trained Language Models (PLMs) often leads to the issue of spurious correlations, which negatively impacts performance, particularly when dealing with out-of-distribution data. To address this problem, we propose SALAD}(Structure Aware and LLM-driven Augmented Data), a novel approach designed to enhance model robustness and generalization by generating structure-aware and counterfactually augmented data for contrastive learning. Our method leverages a tagging-based approach to generate structure-aware positive samples and utilizes large language models (LLMs) to generate counterfactual negative samples with diverse sentence patterns. By applying contrastive learning, SALAD enables the model to focus on learning the structural relationships between key sentence components while minimizing reliance on spurious correlations. We validate our approach through experiments on three tasks: Sentiment Classification, Sexism Detection, and Natural Language Inference. The results demonstrate that SALAD not only improves model robustness and performance across different environments but also enhances generalization to out-of-distribution datasets and cross-domain scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。