针对越南语数据稀缺问题,提出新对比学习框架提升文本表征能力。
ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Tasks
- 基于自然语言推理数据设计监督对比学习方法
- 在5个基准数据集上性能超越PhoBERT,最高提升9.02% F1
- 适合低资源语言的NLU研究与实际应用
高质量文本表征对自然语言理解至关重要,但越南语等低资源语言受限于标注数据不足。尽管预训练模型如PhoBERT和CafeBERT表现良好,其效果仍受数据稀缺制约。对比学习(CL)近年来成为改善句向量表示的有前景方法,使模型能有效区分语义相似与不相似句子。本文提出专为越南语设计的监督对比学习框架ViCLSR,利用现有自然语言推理(NLI)数据集优化句向量表示,并提出适配现有越南语数据集的方法以兼容对比学习。实验表明,ViCLSR在五个基准NLU数据集上显著优于强大的单语预训练模型PhoBERT:ViNLI(+6.97% F1)、ViWikiFC(+4.97% F1)、ViFactCheck(+9.02% F1)、UIT-ViCTSD(+5.36% F1)和ViMMRC2.0(+4.33% Accuracy)。结果证明,监督对比学习可有效缓解越南语等低资源语言的资源瓶颈,提升句向量学习效果。此外,我们深入分析实验结果,揭示对比学习模型优异表现的关键因素。ViCLSR已开源,旨在推动自然语言处理研究进展。
原文摘要 · Abstract (English)
High-quality text representations are crucial for natural language understanding (NLU), but low-resource languages like Vietnamese face challenges due to limited annotated data. While pre-trained models like PhoBERT and CafeBERT perform well, their effectiveness is constrained by data scarcity. Contrastive learning (CL) has recently emerged as a promising approach for improving sentence representations, enabling models to effectively distinguish between semantically similar and dissimilar sentences. We propose ViCLSR (Vietnamese Contrastive Learning for Sentence Representations), a novel supervised contrastive learning framework specifically designed to optimize sentence embeddings for Vietnamese, leveraging existing natural language inference (NLI) datasets. Additionally, we propose a process to adapt existing Vietnamese datasets for supervised learning, ensuring compatibility with CL methods. Our experiments demonstrate that ViCLSR significantly outperforms the powerful monolingual pre-trained model PhoBERT on five benchmark NLU datasets such as ViNLI (+6.97% F1), ViWikiFC (+4.97% F1), ViFactCheck (+9.02% F1), UIT-ViCTSD (+5.36% F1), and ViMMRC2.0 (+4.33% Accuracy). ViCLSR shows that supervised contrastive learning can effectively address resource limitations in Vietnamese NLU tasks and improve sentence representation learning for low-resource languages. Furthermore, we conduct an in-depth analysis of the experimental results to uncover the factors contributing to the superior performance of contrastive learning models. ViCLSR is released for research purposes in advancing natural language processing tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。