用不可逆哈希解决版权文本数据共享难题，确保只授权给合法持有者。
Overcoming Copyright Barriers in Corpus Distribution Through Non-Reversible Hashing
- 通过不可逆哈希将原文本转为密钥，仅允许拥有相同原文的用户匹配标注。
- 在不同版本小说上测试，匹配准确率达98.7%至99.79%。
- 适合需共享受版权保护文本标注的研究者，兼顾合规与实用性。
尽管标注语料库对自然语言处理至关重要，但包含版权内容的语料库难以在研究者间流通。然而，这些语料对全面反映真实场景下的数据多样性必不可少。本文提出一种合法公开分享受版权文学文本标注的方法：语料创建者公开标注及原文的不可逆哈希值；使用者需拥有相同原文，并用相同哈希函数处理自身文本以匹配标注。该方法对用户持有的版权数据版本差异具有鲁棒性。以不同版本小说为例的对齐实验显示，当用户版本足够接近创建者版本时，令牌匹配率可达98.7%至99.79%。本文已公开发布novelshare，一个Python实现工具。
原文摘要 · Abstract (English)
While annotated corpora are crucial in the field of natural language processing (NLP), those containing copyrighted material are difficult to exchange among researchers. Yet, such corpora are necessary to fully represent the diversity of data found in the wild in the context of NLP tasks. We tackle this issue by proposing a method to lawfully and publicly share the annotations of copyrighted literary texts. The corpus creator shares the annotations in clear, along with a non-reversible hashed version of the source material. The corpus user must own the source material, and apply the same hash function to their own tokens, in order to match them to the shared annotations. Crucially, our method is robust to reasonable divergences in the version of the copyrighted data owned by the user. As an illustration, we present alignment experiments on different editions of novels. Our results show that our method is able to correctly align 98.7 to 99.79% of tokens depending on the novel, provided the user version is sufficiently close to the corpus creator's version. We publicly release novelshare, a Python implementation of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。