通过重复序列分析,揭示大模型与自然语言在长程结构上的本质差异。
Repeated Sequences Reveal Gaps between Large Language Models and Natural Language

- 用重复子序列分布和黎尼熵分析文本的长程组织规律。
- 自然语言熵增长稳定,而大模型生成文本的熵指数随规模系统性变化。
- 该方法可量化区分自然语言与大模型输出的深层结构差异。
评估大语言模型(LLMs)是否超越局部流畅性,捕捉自然语言的长程统计结构,仍是开放挑战。现有方法多基于任务性能或短上下文行为,难以揭示生成文本的长程组织特性。本文提出一种基于重复子序列的互补评估框架,通过分析其跨尺度分布并关联高阶黎尼熵,探究有限长度下文本对已有结构的复用机制。在人类写作文本与长度匹配的GPT生成文本上实验发现,尽管幂律模型可描述特定块长范围,但观测到的熵增长更常由对数-幂形式刻画。在各数据集上,自然语言在可及范围内表现出稳定的熵增长模式,个体文本虽有波动,平均行为一致;而GPT生成文本则随模型规模出现系统性且显著的估计指数偏移。结果表明,重复子序列熵可作为定量结构诊断工具,揭示大模型与自然语言在长程组织上的系统性差异,超越表面流畅性层面。
原文摘要 · Abstract (English)
Evaluating whether large language models (LLMs) capture the structure of natural language beyond local fluency remains an open challenge. Existing evaluation methods, largely based on task performance or short-context behavior, provide limited insight into the long-range statistical organization of generated text. We propose a complementary evaluation framework based on repeated subsequences. By analyzing their distribution across scales and relating it to higher-order Rényi entropies, we probe how texts reuse previously established structure under finite-length conditions. Experiments on human-written texts and length-matched GPT-generated texts show that, while power-law models can describe restricted ranges of block length, the observed entropy growth is often equally or better characterized by logarithmic--power forms. Across datasets, natural language exhibits stable entropy-growth patterns over accessible ranges, with consistent average behavior despite variability across individual texts. In contrast, GPT-generated texts show systematic and statistically significant shifts in estimated exponents with model size. These results demonstrate that repeated-subsequence entropy provides a quantitative structural diagnostic that reveals systematic differences in long-range organization, distinguishing natural language from state-of-the-art LLM outputs beyond surface-level fluency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。