用似然方差筛选重要文本，降低模型映射的计算成本
Likelihood Variance as Text Importance for Resampling Texts to Map Language Models
- 根据文本在不同模型间的似然方差加权采样，识别关键文本
- 仅需一半文本量即可达到与均匀采样相当的KL散度估计精度
- 适合需要频繁更新或扩展语言模型对比地图的研究场景
我们解决构建模型映射（model map）的计算开销问题，该映射通过KL散度将多种语言模型嵌入统一空间以进行比较。映射依赖于大量文本上的对数似然值，计算成本与文本数量成正比。为此，我们提出一种重采样方法：根据每个文本在不同模型间对数似然值的方差大小，按比例选取重要文本。实验表明，该方法仅需约一半的文本量即可获得与均匀采样相当的KL散度估计精度，并支持高效地将新模型加入已有映射中。该方法显著提升了语言模型映射的可扩展性与效率。
原文摘要 · Abstract (English)
We address the computational cost of constructing a model map, which embeds diverse language models into a common space for comparison via KL divergence. The map relies on log-likelihoods over a large text set, making the cost proportional to the number of texts. To reduce this cost, we propose a resampling method that selects important texts with weights proportional to the variance of log-likelihoods across models for each text. Our method significantly reduces the number of required texts while preserving the accuracy of KL divergence estimates. Experiments show that it achieves comparable performance to uniform sampling with about half as many texts, and also facilitates efficient incorporation of new models into an existing map. These results enable scalable and efficient construction of language model maps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。