arXiv:2603.04691cs.CL2026-03被引 1

发现停用词分布不符合齐普夫定律,提出基于排名的选词模型。

Non-Zipfian Distribution of Stopwords or Function Words and Subset Selection Models

  • 基于词频排名构建停用词选择概率模型,采用希尔函数描述
  • 该模型能准确解释停用词的贝塔秩函数分布及非停用词的二次拟合现象
  • 适用于语言发育早期或失语症患者等简化语言场景

停用词和功能词在语言中主要起结构作用,信息量较低。与所有词服从齐普夫定律不同,停用词或功能词的词频-排名曲线最佳拟合为贝塔秩函数(BRF)。而非停用词或非功能词的分布也偏离齐普夫定律,但更符合对数词频与对数排名的二次函数关系。基于全词表中停用词的排名,我们提出一个子集选择模型:词被选中的概率是排名 $r$ 的递减希尔函数 $1/(1+(r/r_{mid})^γ)$,不被选中的概率为标准希尔函数 $1/(1+(r_{mid}/r)^γ)$。通过独立文本集合直接估计验证了该模型。分析表明,当全词表服从齐普夫定律时,该模型可导出停用词的BRF分布,并解释非停用词的二次拟合结果。由此推论,齐普夫定律在儿童早期电报式语言或失语症患者中并不成立。

原文摘要 · Abstract (English)

Stopwords and function words are relatively less informative for the content of a language and more often play a structural role in a sentence. Stopwords are ubiquitous words and may contain verbs, adjectives and adverbs. On the other hand, function words are strictly prepositions, conjunctions, pronouns, determiners, qualifiers, articles, interrogatives, and a limited number of auxiliary verbs. In contrast to the well known Zipf's law for rank-frequency plot for all words, the rank-frequency plots for stopwords or function words are best fitted by the Beta Rank Function (BRF). On the other hand, the rank-frequency plots of non-stopwords or non-function-words also deviate from the Zipf's law, but are better described by a quadratic function of log-token-count over log-rank than by BRF. Based on the observed rank of stopwords or function words in the full word list, we propose a stopword/function word/subset selection model that the probability for being selected, as a function of the word's rank $r$, is a decreasing Hill's function ($1/(1+(r/r_{mid})^γ)$); whereas the probability for not being selected is the standard Hill's function ($1/(1+(r_{mid}/r)^γ)$). We validate this selection probability model by a direct estimation from an independent collection of texts. We also show analytically that this model leads to a BRF rank-frequency distribution for stopwords or function words when the original full word list follows the Zipf's law, as well as explaining the quadratic fitting function for the non-stopwords or non-function-words. A corollary of these results is that Zipf's law is not expected to be true for telegraphic speech in early childhood language learners or in agrammatism patients.

语言模型停用词统计规律语言发育

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。