用对比去噪增强BERT对词语替换等扰动的鲁棒性
CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising
- 设计轻量级对比去噪自编码器，联合优化对比与重构目标
- 在多种扰动下嵌入相似度提升，强扰动时效果更显著
- 适合需要稳定语义表示的场景，如信息检索、文本匹配
预训练语言模型虽显著提升了句子表征学习能力，但其嵌入仍对语义保持型文本扰动（如同义词替换、掩码和词丢弃）敏感。本文提出一种轻量级对比去噪自编码器（CDAE），通过联合优化对比与重建目标，对预训练BERT嵌入进行精炼，以学习扰动不变的表示。我们在多种强度的扰动策略下评估该框架，并与原始BERT嵌入及SimCSE进行对比。实验结果表明，CDAE在各类扰动下均能保持更高嵌入相似度，且随着扰动强度增加，改进效果更为明显，有效增强了表示稳定性并保留了语义信息，凸显了扰动不变学习在提升句子嵌入方面的潜力。代码已公开于：https://github.com/ComputationIASBS/CDAE
原文摘要 · Abstract (English)
Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embedding by jointly optimizing contrastive and reconstruction objective to learn perturbation-invariant representation. We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。