用伽马混合模型刻画小模型语义相似度分布，提升对齐精度。
Gamma Mixture Modeling for Cosine Similarity in Small Language Models
- 基于主题聚类的层次结构自然生成伽马混合相似度
- 实测显示相似度分布可被截断伽马分布精准拟合
- 提供可落地的期望最大化算法，适用于模型评估
我们研究了句子嵌入的余弦相似度，发现其分布可用伽马混合模型良好刻画。从固定语料库中，计算所有文档嵌入与参考查询嵌入之间的相似度。实验表明，这些分布常能被一个在[-1,1]区间内平移并截断的伽马分布所拟合，在多数情况下则适合用伽马混合模型描述。我们提出一种启发式模型：主题的层次聚类会自然导致相似度分数呈现伽马混合结构。最后，我们设计了一种用于拟合平移伽马混合模型的期望最大化算法，为建模相似度分布提供了实用工具。
原文摘要 · Abstract (English)
We study the cosine similarity of sentence transformer embeddings and observe that they are well modeled by gamma mixtures. From a fixed corpus, we measure similarities between all document embeddings and a reference query embedding. Empirically we find that these distributions are often well captured by a gamma distribution shifted and truncated to [-1,1], and in many cases, by a gamma mixture. We propose a heuristic model in which a hierarchical clustering of topics naturally leads to a gamma-mixture structure in the similarity scores. Finally, we outline an expectation-maximization algorithm for fitting shifted gamma mixtures, which provides a practical tool for modeling similarity distributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。