用字形簇替代字符编码点，更准确评估多语言文本错误。
grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP
- 以字形簇为单位计算文本相似度，避免多编码导致的误差。
- 在泰米尔语和僧伽罗语中实现精准字形识别与拆分组合。
- 适合处理复杂书写系统，如印度文字，提升OCR评估可靠性。
现有词汇距离、相似度及评估指标基于Unicode码点运行，但在一个字形由多个码点表示的书写系统中，可能错误地反映文本错误。我们提出grapheme-kit，一个开源Python库，将这些指标扩展至字形簇层面。该库还提供了对泰米尔语和僧伽罗语的改进字形处理功能，包括准确的字形簇识别以及字形组合/分解工具。通过光学字符识别（OCR）案例研究，我们证明字形级度量能更真实地评估复杂脚本的性能。
原文摘要 · Abstract (English)
Existing lexical distance, similarity, and evaluation metrics operate on Unicode code points, which can misrepresent errors in writing systems where a single grapheme is represented by multiple Unicode code points. We introduce grapheme-kit, an open-source Python library that extends these metrics to operate on grapheme clusters instead. The library also provides improved grapheme processing for Tamil and Sinhala, including accurate grapheme cluster identification and grapheme composition/decomposition utilities. Through an OCR case study, we demonstrate that grapheme-level metrics provide a more faithful evaluation of complex scripts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。