用分布图分析文本,揭示生成模型的隐藏统计特征。
DMAP: A Distribution Map for Text
- 将文本映射到单位区间样本,同时保留概率与排名信息
- 在消费级硬件上高效计算,支持多种文本分析应用
- 适合研究生成文本检测、数据完整性验证等场景
大型语言模型(LLMs)为文本统计分析提供了强大工具,其生成的下一个词概率分布蕴含丰富信息。传统方法如困惑度(perplexity)无法充分考虑上下文;对某个预测概率的解读依赖于条件分布形状所编码的合理选择数量。本文提出DMAP,一种数学严谨的方法,通过语言模型将文本映射为单位区间内的样本集合,联合编码排序与概率信息。该表示支持高效、模型无关的分析,并适用于多种应用场景。我们通过三个案例验证其有效性:(i) 验证生成参数以确保数据完整性;(ii) 探讨概率曲率在机器生成文本检测中的作用;(iii) 通过法医分析揭示下游模型在经过合成数据微调后留下的统计痕迹。结果表明,DMAP提供了一种简单易算、广泛适用且统一的文本统计视图,为未来基于LLM的文本分析研究奠定基础。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are a powerful tool for statistical text analysis, with derived sequences of next-token probability distributions offering a wealth of information. Extracting this signal typically relies on metrics such as perplexity, which do not adequately account for context; how one should interpret a given next-token probability is dependent on the number of reasonable choices encoded by the shape of the conditional distribution. In this work, we present DMAP, a mathematically grounded method that maps a text, via a language model, to a set of samples in the unit interval that jointly encode rank and probability information. This representation enables efficient, model-agnostic analysis and supports a range of applications. We illustrate its utility through three case studies: (i) validation of generation parameters to ensure data integrity, (ii) examining the role of probability curvature in machine-generated text detection, and (iii) a forensic analysis revealing statistical fingerprints left in downstream models that have been subject to post-training on synthetic data. Our results demonstrate that DMAP offers a unified statistical view of text that is simple to compute on consumer hardware, widely applicable, and provides a foundation for further research into text analysis with LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。