测试大模型学生僻词的能力，发现模型在罕见词上表现很差。
LongTail-Swap: benchmarking language models' abilities on rare words
- 设计新评测集，聚焦罕见词的语义和句法使用。
- 16个模型在罕见词上平均表现低于头部词汇30%以上。
- 揭示不同架构在罕见词泛化能力上的显著差异，适合研究小样本学习者。
儿童以少量数据就能学会说话，并能通过少量示例掌握新词，表现出极高的数据效率。现有BabyLM挑战主要关注词汇分布头部的性能，忽略了长尾部分。本文提出LongTail-Swap（LT-Swap）基准，专门评估语言模型在极低暴露下学习罕见词的能力，模拟婴儿习得过程。该基准基于特定预训练语料库构建，包含可接受与不可接受的句子对，用于隔离罕见词的语义与句法用法。模型在零样本条件下通过计算每对句子的平均对数概率进行评估。我们构建了两个测试集，分别对应10M和100M词的BabyLM训练集，评估了16个来自BabyLM排行榜的模型。结果表明，语言模型在罕见词上的表现普遍较差，且不同架构间的差距在长尾部分远大于头部。这为理解哪些架构更擅长罕见词泛化提供了新视角。代码已公开。
原文摘要 · Abstract (English)
Children learn to speak with a low amount of data and can be taught new words on a few-shot basis, making them particularly data-efficient learners. The BabyLM challenge aims at exploring language model (LM) training in the low-data regime but uses metrics that concentrate on the head of the word distribution. Here, we introduce LongTail-Swap (LT-Swap), a benchmark that focuses on the tail of the distribution, i.e., measures the ability of LMs to learn new words with very little exposure, like infants do. LT-Swap is a pretraining corpus-specific test set of acceptable versus unacceptable sentence pairs that isolate semantic and syntactic usage of rare words. Models are evaluated in a zero-shot fashion by computing the average log probabilities over the two members of each pair. We built two such test sets associated with the 10M words and 100M words BabyLM training sets, respectively, and evaluated 16 models from the BabyLM leaderboard. Our results not only highlight the poor performance of language models on rare words but also reveal that performance differences across LM architectures are much more pronounced in the long tail than in the head. This offers new insights into which architectures are better at handling rare word generalization. We've also made the code publicly avail
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。