用对数似然向量构建模型映射，直观比较不同语言模型的响应分布。
Language Model Maps for Prompt-Response Distributions via Log-Likelihood Vectors
- 通过提示-响应对的对数似然向量构建模型空间，距离近似表示KL散度。
- 映射能捕捉模型属性、任务性能及提示修改带来的系统性变化。
- 引入互信息向量降低无条件分布干扰，更反映训练数据差异。
我们提出一种方法，通过提示-响应对的对数似然向量表示语言模型，并构建模型映射以比较其条件分布。在此空间中，模型间的距离近似于对应条件分布之间的KL散度。在大量公开语言模型上的实验表明，该映射能捕捉有意义的全局结构，包括模型属性与任务性能的关系。方法还能识别提示修改引起的系统性变化及其近似可加性，为分析和预测复合提示操作的效果提供途径。此外，我们引入点互信息（PMI）向量以减少无条件分布的影响；在某些情况下，基于PMI的模型映射更能反映与训练数据相关的差异。整体框架支持对输入依赖型模型行为的分析。
原文摘要 · Abstract (English)
We propose a method that represents language models by log-likelihood vectors over prompt-response pairs and constructs model maps for comparing their conditional distributions. In this space, distances between models approximate the KL divergence between the corresponding conditional distributions. Experiments on a large collection of publicly available language models show that the maps capture meaningful global structure, including relationships to model attributes and task performance. The method also captures systematic shifts induced by prompt modifications and their approximate additive compositionality, suggesting a way to analyze and predict the effects of composite prompt operations. We further introduce pointwise mutual information (PMI) vectors to reduce the influence of unconditional distributions; in some cases, PMI-based model maps better reflect training-data-related differences. Overall, the framework supports the analysis of input-dependent model behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。