仅用一次前向传播实现高效无监督句向量学习,适合大模型场景。
CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass
- 利用生成式模型结构特性,单次前向传播完成对比学习
- 相比现有方法,训练耗时与内存占用显著降低
- 提出双比率评估指标,更可靠衡量语义分布质量
句子表示是信息检索与计算语言学中的基础任务,对文本聚类、内容分析、问答系统和网络搜索等应用具有重要意义。近年来,预训练语言模型(PLMs)推动了该领域的显著进展,尤其体现在基于判别式PLM(如BERT)的无监督嵌入方法。然而,受时间和计算资源限制,鲜有研究将无监督句向量学习与参数规模更大的生成式PLM结合。鉴于学术界和工业界主流模型均为生成式架构,亟需一种面向解码器单一结构的高效无监督文本表示框架。为此,本文提出CSE-SFP,创新性地利用生成式模型的结构特性,仅通过一次前向传播即可实现有效的无监督对比学习。大量实验表明,CSE-SFP不仅生成更高质量的嵌入,还显著降低训练时间与内存消耗。此外,我们引入两个比例度量,联合评估对齐性与均匀性,为编码模型的语义空间特性提供更稳健的评估方式。
原文摘要 · Abstract (English)
As a fundamental task in Information Retrieval and Computational Linguistics, sentence representation has profound implications for a wide range of practical applications such as text clustering, content analysis, question-answering systems, and web search. Recent advances in pre-trained language models (PLMs) have driven remarkable progress in this field, particularly through unsupervised embedding derivation methods centered on discriminative PLMs like BERT. However, due to time and computational constraints, few efforts have attempted to integrate unsupervised sentence representation with generative PLMs, which typically possess much larger parameter sizes. Given that state-of-the-art models in both academia and industry are predominantly based on generative architectures, there is a pressing need for an efficient unsupervised text representation framework tailored to decoder-only PLMs. To address this concern, we propose CSE-SFP, an innovative method that exploits the structural characteristics of generative models. Compared to existing strategies, CSE-SFP requires only a single forward pass to perform effective unsupervised contrastive learning. Rigorous experimentation demonstrates that CSE-SFP not only produces higher-quality embeddings but also significantly reduces both training time and memory consumption. Furthermore, we introduce two ratio metrics that jointly assess alignment and uniformity, thereby providing a more robust means for evaluating the semantic spatial properties of encoding models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。