提出一种同时保持词频和长程相关性的语言模拟方法。
A Zipf-preserving, long-range correlated surrogate for written language and other symbolic sequences
- 用分数高斯噪声映射到真实词频分布生成新序列
- 在英文和拉丁文上验证,长程相关性指数与原序列一致
- 适合研究语言、基因组等符号系统的结构起源
书写语言和基因组DNA等符号序列具有典型的频率分布和跨越数千符号的长程相关性。语言中表现为词频的齐普夫定律及跨数百至数千词元的持续相关性,基因组则体现为碱基组成和嘌呤-嘧啶映射下的长记忆随机游走。现有代理模型通常只能保留频率或相关性之一,无法兼顾。本文提出一种新方法,能同时保持原始序列的符号频率和长程相关结构(以去趋势波动分析DFA指数量化)。通过将分数高斯噪声(FGN)按频率保持方式映射到经验直方图,生成符合原始序列一阶统计量和长程标度特性的代理序列,同时打乱短程依赖。在英、拉丁语代表性文本上验证有效,并拓展至基因组DNA,成功复现碱基组成与DFA标度行为。该方法为解析符号系统结构特征、检验跨语言、基因组等领域标度律与记忆效应的成因提供了严谨工具。
原文摘要 · Abstract (English)
Symbolic sequences such as written language and genomic DNA display characteristic frequency distributions and long-range correlations extending over many symbols. In language, this takes the form of Zipf's law for word frequencies together with persistent correlations spanning hundreds or thousands of tokens, while in DNA it is reflected in nucleotide composition and long-memory walks under purine-pyrimidine mappings. Existing surrogate models usually preserve either the frequency distribution or the correlation properties, but not both simultaneously. We introduce a surrogate model that retains both constraints: it preserves the empirical symbol frequencies of the original sequence and reproduces its long-range correlation structure, quantified by the detrended fluctuation analysis (DFA) exponent. Our method generates surrogates of symbolic sequences by mapping fractional Gaussian noise (FGN) onto the empirical histogram through a frequency-preserving assignment. The resulting surrogates match the original in first-order statistics and long-range scaling while randomising short-range dependencies. We validate the model on representative texts in English and Latin, and illustrate its broader applicability with genomic DNA, showing that base composition and DFA scaling are reproduced. This approach provides a principled tool for disentangling structural features of symbolic systems and for testing hypotheses on the origin of scaling laws and memory effects across language, DNA, and other symbolic domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。