发现人类语言编码的最优性新类别，解释了齐普夫定律的起源。
On the class of coding optimality of human languages and the origins of Zipf's law
- 提出一类线性偏离最优编码的系统，可自然导出齐普夫定律。
- 人类语言在频率-等级双对数图中呈直线，表明编码接近最优。
- 该理论可检验压缩系统中齐普夫定律的产生条件，适合语言学与信息论研究者。
本文提出一类新的编码系统最优性。此类系统相对于最优编码呈线性偏移，因而表现出齐普夫定律——即频率秩的幂律分布。在此类系统中，齐普夫定律、大小-秩定律和大小-概率定律构成类似群的结构。我们识别出属于该类别的语言，所有与齐普夫定律足够吻合的语言均可能是该类成员。相比之下，其他物种的通信系统因呈现指数分布而无法归属此类，但海豚和座头鲸可能例外。本文提供对频率-等级双对数图的新见解：任何系统在该坐标系中若呈直线，表明非奇异编码与唯一可解编码下的最优码长存在线性偏差，其斜率即为齐普夫定律的指数。对于受唯一可解性约束的压缩系统，此类直线可能意味着系统接近编码最优。本文支持压缩是齐普夫定律起源的假说，并给出可检验的涌现条件。
原文摘要 · Abstract (English)
Here we present a new class of optimality for coding systems. Members of that class are displaced linearly from optimal coding and thus exhibit Zipf's law, namely a power-law distribution of frequency ranks. Within that class, Zipf's law, the size-rank law and the size-probability law form a group-like structure. We identify human languages that are members of the class. All languages showing sufficient agreement with Zipf's law are potential members of the class. In contrast, there are communication systems in other species that cannot be members of that class for exhibiting an exponential distribution instead but dolphins and humpback whales might. We provide a new insight into plots of frequency versus rank in double logarithmic scale. For any system, a straight line in that scale indicates that the lengths of optimal codes under non-singular coding and under uniquely decodable encoding are displaced by a linear function whose slope is the exponent of Zipf's law. For systems under compression and constrained to be uniquely decodable, such a straight line may indicate that the system is coding close to optimality. We provide support for the hypothesis that Zipf's law originates from compression and define testable conditions for the emergence of Zipf's law in compressing systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。