用量子相对熵绘制1101个句子编码器的分布图，揭示其相似性与任务表现关系。
Map of Encoders -- Mapping Sentence Encoders using Quantum Relative Entropy
- 以嵌入矩阵和内积矩阵构建编码器特征，计算其与基准编码器的量子相对熵。
- 地图显示相似属性编码器在空间上邻近，且可准确预测检索与聚类性能。
- 为大规模编码器比较提供可视化工具，适合模型选型与研究者参考。
我们提出一种方法，通过创建编码器地图来大规模比较和可视化句子编码器，其中每个编码器均相对于其他编码器进行表示。具体而言，首先利用句子集合的嵌入矩阵表示一个编码器，每行对应一个句子的嵌入；接着基于该嵌入矩阵计算配对内积（PIP）矩阵；最后，为每个编码器生成一个特征向量，反映其相对于单位基准编码器的量子相对熵（QRE）。我们构建了涵盖1101个公开可用句子编码器的地图，为预训练句子编码器的格局提供了新视角。该地图准确反映了编码器间的多种关系，具有相似属性的编码器在地图上位置相近。此外，我们的编码器特征向量可准确推断下游任务性能，如检索和聚类任务，证明了地图的忠实性。
原文摘要 · Abstract (English)
We propose a method to compare and visualise sentence encoders at scale by creating a map of encoders where each sentence encoder is represented in relation to the other sentence encoders. Specifically, we first represent a sentence encoder using an embedding matrix of a sentence set, where each row corresponds to the embedding of a sentence. Next, we compute the Pairwise Inner Product (PIP) matrix for a sentence encoder using its embedding matrix. Finally, we create a feature vector for each sentence encoder reflecting its Quantum Relative Entropy (QRE) with respect to a unit base encoder. We construct a map of encoders covering 1101 publicly available sentence encoders, providing a new perspective of the landscape of the pre-trained sentence encoders. Our map accurately reflects various relationships between encoders, where encoders with similar attributes are proximally located on the map. Moreover, our encoder feature vectors can be used to accurately infer downstream task performance of the encoders, such as in retrieval and clustering tasks, demonstrating the faithfulness of our map.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。