用熵值差异检测两组文本是否相似,适合大规模文本分析。
A Two-Sample Test of Text Generation Similarity
- 基于语言模型估算文本熵,构建双样本检验框架。
- 理论证明统计量渐近正态,多数据分拆提升检验功效。
- 适用于需区分文档类别的场景,如舆情分析、文献分类。
随着数字化文本数据的激增,对文本模式进行可靠推断的需求日益迫切。本文提出一种新颖的双样本文本检验方法,用于比较两组文档之间的相似性。其核心假设是:生成文本数据的概率映射在两组文档间是否相同。该方法通过比较文档的熵来评估文本相似性,熵值利用基于神经网络的语言模型进行估计。测试统计量基于估计算法与推断框架构建:先在估计集上近似熵,再在剩余数据集上进行推断。理论上,在较弱条件下,该统计量渐近服从正态分布。为增强检验功效,提出了多数据分拆策略,将多个p值合并为统一决策。模拟研究与真实数据案例表明,所提方法在保持名义第一类错误率的同时,相较于现有方法具有更高的检验功效。该方法为判断文档类别差异提供了新工具,尤其适用于大规模文本信息至关重要的领域。
原文摘要 · Abstract (English)
The surge in digitized text data requires reliable inferential methods on observed textual patterns. This article proposes a novel two-sample text test for comparing similarity between two groups of documents. The hypothesis is whether the probabilistic mapping generating the textual data is identical across two groups of documents. The proposed test aims to assess text similarity by comparing the entropy of the documents. Entropy is estimated using neural network-based language models. The test statistic is derived from an estimation-and-inference framework, where the entropy is first approximated using an estimation set, followed by inference on the remaining data set. We showed theoretically that under mild conditions, the test statistic asymptotically follows a normal distribution. A multiple data-splitting strategy is proposed to enhance test power, which combines p-values into a unified decision. Various simulation studies and a real data example demonstrated that the proposed two-sample text test maintains the nominal Type one error rate while offering greater power compared to existing methods. The proposed method provides a novel solution to assert differences in document classes, particularly in fields where large-scale textual information is crucial.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。