无需训练，通过重复机制增强大模型嵌入效果
Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition
- 利用文本重复触发反向注意力，增强上下文编码
- 在C-MTEB上多任务零样本性能显著提升
- 适合追求零样本能力提升的模型优化者
语言模型可视为将文本映射到欧氏空间的函数，嵌入向量质量直接影响模型表现，而神经网络训练过程存在诸多不确定性。本文提出一种无需额外训练的新型反向注意力机制，旨在提升预训练语言模型在零样本场景下的性能。该方法通过文本重复激发模型对上下文信息的捕捉能力，显著增强了嵌入表示。在中文大规模文本嵌入基准（C-MTEB）上的实验表明，该方法在多个任务中均取得显著提升，为增强零样本学习能力提供了有效路径。
原文摘要 · Abstract (English)
Language models can be viewed as functions that embed text into Euclidean space, where the quality of the embedding vectors directly determines model performance, training such neural networks involves various uncertainties. This paper focuses on improving the performance of pre-trained language models in zero-shot settings through a simple and easily implementable method. We propose a novel backward attention mechanism to enhance contextual information encoding. Evaluated on the Chinese Massive Text Embedding Benchmark (C-MTEB), our approach achieves significant improvements across multiple tasks, providing valuable insights for advancing zero-shot learning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。