发现kNN语言模型对低频词预测无帮助，反而只提升高频词表现。
Long-Tail Crisis in Nearest Neighbor Language Models
- 通过分析数据存储中词频分布与检索精度，研究低频词预测机制
- 实验表明模型在低频目标词上未改善概率估计，仅提升高频词性能
- 揭示了长尾现象下kNN-LM的局限性，适合关注高频词优化的研究者
k最近邻语言模型（kNN-LM）是一种检索增强型语言模型，通过推理时直接访问由任意文本数据构建的大规模数据存储来降低给定文本的困惑度。人们普遍认为其成功源于显式记忆——即数据存储，能增强对长尾现象的预测能力。然而，以往研究主要验证了其在长尾上下文上的检索能力，而对模型在推理过程中对长尾目标词的概率估计性能仍缺乏深入探索。本文系统研究了kNN-LM在低频词上的行为，考察了预测概率、检索准确率、数据存储中的词频分布以及产品量化带来的近似误差。实验结果表明，kNN-LM并未提升低频词的预测性能，其优势主要集中在高频词上，无论数据存储中是否存在长尾上下文。
原文摘要 · Abstract (English)
The $k$-nearest-neighbor language model ($k$NN-LM), one of the retrieval-augmented language models, improves the perplexity for given text by directly accessing a large datastore built from any text data during inference. A widely held hypothesis for the success of $k$NN-LM is that its explicit memory, i.e., the datastore, enhances predictions for long-tail phenomena. However, prior works have primarily shown its ability to retrieve long-tail contexts, leaving the model's performance remain underexplored in estimating the probabilities of long-tail target tokens during inference. In this paper, we investigate the behavior of $k$NN-LM on low-frequency tokens, examining prediction probability, retrieval accuracy, token distribution in the datastore, and approximation error of the product quantization. Our experimental results reveal that $k$NN-LM does not improve prediction performance for low-frequency tokens but mainly benefits high-frequency tokens regardless of long-tail contexts in the datastore.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。