模型表征越分散，文本预测能力越强，可无监督提升性能
On the Predictive Power of Representation Dispersion in Language Models
- 用隐藏向量间余弦距离衡量表征分散度
- 分散度越高，困惑度越低，跨模型跨领域均成立
- 无需标注数据即可识别难样本、优化检索层、改进训练
我们发现语言模型的文本预测能力与其嵌入空间的广度密切相关：表征分布越广的模型，困惑度越低。具体而言，在多种模型家族（LLaMA、Qwen等）和领域（维基百科、新闻、科学摘要）中，表征分散度（隐藏向量间的平均成对余弦距离）与困惑度呈强烈负相关。除了揭示这一关联，我们还展示了分散度在无标签数据下的多种实用价值：首先，通过测量未标注文本的分散度，可按难度排序样本并识别新领域的难点片段，实现高效的数据筛选；其次，高分散度的层能精准定位最佳检索表示，替代逐层搜索；最后，引入简单的“推离”目标函数，可在单域和跨域场景中提升分散度，并直接降低困惑度。代码已公开于https://github.com/yanhong-lbh/rep_dispersion。
原文摘要 · Abstract (English)
We show that a language model's ability to predict text is tightly linked to the breadth of its embedding space: models that spread their contextual representations more widely tend to achieve lower perplexity. Concretely, we find that representation dispersion--the average pairwise cosine distance among hidden vectors--strongly and negatively correlates with perplexity across diverse model families (LLaMA, Qwen, and others) and domains (Wikipedia, news, scientific abstracts). Beyond illustrating this link, we show how dispersion can be leveraged for a range of practical tasks--without requiring labeled data. First, measuring dispersion on unlabeled text allows us to rank examples by difficulty and identify hard slices in new domains, offering a data-efficient tool for screening and prioritizing models before full evaluation. Next, we find that identifying layers with higher dispersion pinpoints the best representations for retrieval-based methods such as kNN-LM, bypassing exhaustive layer-by-layer searches. Finally, we integrate a simple "push-away" objective into training, which increases dispersion in both single-domain and cross-domain scenarios and directly improves perplexity in each. Code is available at https://github.com/yanhong-lbh/rep_dispersion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。