让大模型双向看世界，提升语义表征能力
Learning to Look at the Other Side: A Semantic Probing Study of Word Embeddings in LLMs with Enabled Bidirectional Attention
- 在Llama模型中逐步开启双向注意力机制
- 对比学习增强语义表征，显著提升探针任务表现
- 适合关注模型内部表征与注意力机制的研究者
自回归大语言模型在语言理解和生成方面表现出色，但在文本嵌入任务和探针分析中的应用进展较慢，这主要受限于单向注意力机制。本文旨在探究通过开启双向注意力是否可突破这一限制。我们通过对Llama架构的不同变体进行额外训练，逐步启用双向注意力以及无监督/有监督的对比学习，以检验其对语义表示的影响。
原文摘要 · Abstract (English)
Autoregressive Large Language Models (LLMs) demonstrate exceptional performance in language understanding and generation. However, their application in text embedding tasks has been relatively slow, along with the analysis of their semantic representation in probing tasks, due to the constraints of the unidirectional attention mechanism. This paper aims to explore whether such constraints can be overcome by enabling bidirectional attention in LLMs. We tested different variants of the Llama architecture through additional training steps, progressively enabling bidirectional attention and unsupervised/supervised contrastive learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。