高效检测网页语料中是否存在文本拷贝，适合版权验证。
FindMyText: Robust, Scalable Detection of Text Containment in Large Web-Crawled Corpora
- 通过指纹链匹配技术，精准识别近似原文的复制内容。
- 在ArXiv、Wikipedia等三类数据集上性能优于现有方法。
- 支持大规模分布式索引，可处理海量网页文本。
我们提出FindMyText，一个开源Python工具，用于高效判断给定文本是否在文本语料库中以完整或部分形式存在。该工具基于文档指纹技术，创新性地引入了匹配指纹序列的显式机制，通过识别指纹链，能更可靠地检测近似原文的复制内容，而非仅依赖文本相似性。这使其特别适用于验证语料库中是否存在受版权保护的内容。系统采用分布式、基于磁盘的索引框架，可扩展至大规模网络爬取数据集。通过新构建的文本包含性评估基准，我们在三个数据集（ArXiv论文、Wikipedia、通用网页内容）上证明，FindMyText在各项指标上均优于其他方法。
原文摘要 · Abstract (English)
We present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus. The tool builds on prior techniques for document fingerprinting, but extends them with a novel mechanism to explicitly capture sequences of matching fingerprints. By identifying such chains, the tool can more reliably detect near-verbatim copies of a given text rather than mere textual similarities. This makes FindMyText particularly suited for verifying the presence of copyrighted material in a corpus. Leveraging a distributed, disk-based indexing framework, the system scales to large web-crawled datasets. Using a new benchmark for evaluating text containment methods, we show that FindMyText outperforms alternative approaches across three datasets (ArXiv papers, Wikipedia, and generic web content).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。