通过词元分析自动识别文本中实体的相似性，实现可解释的跨文档对比。
Explainable identification of similarities between entities for discovery in large text
- 基于n-gram频率加权，结合语言通用性惩罚机制计算文本相似性。
- 在传记、科学文献等多领域验证，可有效发现人工难辨的实体关联。
- 方法无需参数调优，结果可可视化，适合需要透明推理的场景。
数字化文本数量庞大，自动文本比较对提取人工难以发现的深层洞察至关重要。现有AI工具常仅关注文本表面相似性，而非其所讨论实体的真实相似性。本文提出一种非参数化的n-gram分析框架，通过加权公式评估词元重要性：当词元在两篇文档中同时高频出现时得分更高，但若在英语中普遍高频则被惩罚。结合词云等可视化工具，该方法能清晰呈现相似模式。实验表明，该框架在传记、科学文献、历史文本等领域均能有效揭示实体间可解释的相似性，提供确定性解决方案。代码已公开。
原文摘要 · Abstract (English)
With the availability of virtually infinite number text documents in digital format, automatic comparison of textual data is essential for extracting meaningful insights that are difficult to identify manually. Many existing tools, including AI and large language models, struggle to provide precise and explainable insights into textual similarities. In many cases they determine the similarity between documents as reflected by the text, rather than the similarities between the subjects being discussed in these documents. This study addresses these limitations by developing an n-gram analysis framework designed to compare documents automatically and uncover explainable similarities. A scoring formula is applied to assigns each of the n-grams with a weight, where the weight is higher when the n-grams are more frequent in both documents, but is penalized when the n-grams are more frequent in the English language. Visualization tools like word clouds enhance the representation of these patterns, providing clearer insights. The findings demonstrate that this framework effectively uncovers similarities between text documents, offering explainable insights that are often difficult to identify manually. This non-parametric approach provides a deterministic solution for identifying similarities across various fields, including biographies, scientific literature, historical texts, and more. Code for the method is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。