arXiv:2504.04131cs.CL2025-04被引 1

专为法律文本设计的高精度分句工具,提升大规模法律检索的准确性。

Precise Legal Sentence Boundary Detection for Retrieval at Scale: NUPunkt and CharBoundary

  • 针对法律文本的缩写和复杂结构,优化了分句算法
  • NUPunkt达91.1%精确率,每秒处理1000万字符
  • 适合需要高精度法律检索的系统,如尽职调查与电子取证

我们提出 NUPunkt 与 CharBoundary 两款句子边界检测工具,专为大规模法律文本处理优化,适用于尽职调查、电子取证和法律研究等场景。这些工具解决法律文档中专业引用、缩写和复杂句式带来的挑战,其在五个涵盖超过25,000份文件、197,000个标注句界的法律数据集上的实验表明,NUPunkt 在保持432 MB内存占用的前提下实现91.1%的精确率,每秒处理1000万字符;CharBoundary 模型提供可调的精确率-召回率平衡,大模型取得最高F1值0.782。相比通用工具,NUPunkt 精确率提升29%-32%,多百万文档集合可在分钟级完成处理。两者均仅需标准CPU,无需专用加速器。NUPunkt 为纯Python实现,无外部依赖;CharBoundary 依赖scikit-learn,支持ONNX运行时优化。两款工具均开源于MIT协议,可通过PyPI安装,并在https://sentences.aleainstitute.ai/ 上交互测试。通过减少上下文碎片化,每提高1%精确率即显著改善检索增强生成系统的推理质量。

原文摘要 · Abstract (English)

We present NUPunkt and CharBoundary, two sentence boundary detection libraries optimized for high-precision, high-throughput processing of legal text in large-scale applications such as due diligence, e-discovery, and legal research. These libraries address the critical challenges posed by legal documents containing specialized citations, abbreviations, and complex sentence structures that confound general-purpose sentence boundary detectors. Our experimental evaluation on five diverse legal datasets comprising over 25,000 documents and 197,000 annotated sentence boundaries demonstrates that NUPunkt achieves 91.1% precision while processing 10 million characters per second with modest memory requirements (432 MB). CharBoundary models offer balanced and adjustable precision-recall tradeoffs, with the large model achieving the highest F1 score (0.782) among all tested methods. Notably, NUPunkt provides a 29-32% precision improvement over general-purpose tools while maintaining exceptional throughput, processing multi-million document collections in minutes rather than hours. Both libraries run efficiently on standard CPU hardware without requiring specialized accelerators. NUPunkt is implemented in pure Python with zero external dependencies, while CharBoundary relies only on scikit-learn and optional ONNX runtime integration for optimized performance. Both libraries are available under the MIT license, can be installed via PyPI, and can be interactively tested at https://sentences.aleainstitute.ai/. These libraries address critical precision issues in retrieval-augmented generation systems by preserving coherent legal concepts across sentences, where each percentage improvement in precision yields exponentially greater reductions in context fragmentation, creating cascading benefits throughout retrieval pipelines and significantly enhancing downstream reasoning quality.

法律AI句子分割高精度检索增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。