测试RWKV生成句子嵌入效果，发现其零样本表现不如传统GloVe。
Exploring RWKV for Sentence Embeddings: Layer-wise Analysis and Baseline Comparison for Semantic Similarity
- 分层分析RWKV模型各层的语义表征能力
- 在MRPC数据集上Spearman相关性低于GloVe基线
- 虽计算效率高但语义相似度任务需进一步优化
本文研究了新型语言模型架构RWKV在零样本场景下生成句子嵌入的有效性。通过分层分析预训练RWKV模型不同隐藏层的语义相似度表征能力，在Microsoft Research Paraphrase Corpus（MRPC）数据集上使用Spearman相关性进行评估，并与基于GloVe的基线方法对比。结果表明，尽管RWKV嵌入能捕捉部分语义相关性，但在Spearman相关性指标上仍逊于GloVe基线。同时分析了推理时间与GPU内存占用，揭示了其在计算效率上的权衡。研究提示，虽然RWKV具备线性扩展优势，但其在语义相似度任务中的零样本嵌入质量仍需进一步探究，可能需要针对特定任务进行微调以超越简单基线。
原文摘要 · Abstract (English)
This paper investigates the efficacy of RWKV, a novel language model architecture known for its linear attention mechanism, for generating sentence embeddings in a zero-shot setting. I conduct a layer-wise analysis to evaluate the semantic similarity captured by embeddings from different hidden layers of a pre-trained RWKV model. The performance is assessed on the Microsoft Research Paraphrase Corpus (MRPC) dataset using Spearman correlation and compared against a GloVe-based baseline. My results indicate that while RWKV embeddings capture some semantic relatedness, they underperform compared to the GloVe baseline in terms of Spearman correlation. I also analyze the inference time and GPU memory usage, highlighting the computational trade-offs associated with RWKV embeddings. The findings suggest that while RWKV offers potential advantages in terms of linear scaling, its zero-shot sentence embedding quality for semantic similarity tasks requires further investigation and potential task-specific fine-tuning to match or exceed simpler baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。