开源越南语文本规范化工具,专为语音合成与自然语言处理设计。
VietNormalizer: An Open-Source, Dependency-Free Python Library for Vietnamese Text Normalization in TTS and NLP Applications
- 基于规则的统一流程,无需依赖神经网络
- 支持数字、日期、货币、缩写等7类文本转换
- 零依赖安装,适合低资源语言处理场景
我们提出 VietNormalizer1,一个面向语音合成(TTS)和自然语言处理(NLP)应用的开源、无依赖越南语文本规范化库。真实越南语文本中充斥着大量非标准词汇(NSWs),如数字、日期、时间、货币、百分比、缩写及外来词,必须转换为可读的越南语发音形式才能用于语音合成或下游处理。现有工具或依赖复杂神经模型且覆盖范围窄,或嵌入大型NLP工具包无法独立安装。VietNormalizer通过统一规则管道实现:(1)将整数、小数及大数转为越南语文字;(2)日期时间转为口语化表达;(3)处理越南盾和美元金额;(4)展开百分比;(5)通过自定义CSV字典解析缩写;(6)将非越南语借词音近转写为越南语发音;(7)执行Unicode归一化并移除表情符号与特殊字符。所有正则表达式在初始化时预编译,支持高吞吐批量处理,内存开销极低,无需GPU或外部API。可通过 pip install vietnormalizer 安装,发布于PyPI与GitHub(https://github.com/nghimestudio/vietnormalizer),采用MIT许可。本文讨论设计选择、现有方法局限性,并探讨规则范式在其他低资源声调与黏着语中的泛化潜力。
原文摘要 · Abstract (English)
We present VietNormalizer1, an open-source, zero-dependency Python library for Vietnamese text normalization targeting Text-to-Speech (TTS) and Natural Language Processing (NLP) applications. Vietnamese text normalization is a critical yet underserved preprocessing step: real-world Vietnamese text is densely populated with non-standard words (NSWs), including numbers, dates, times, currency amounts, percentages, acronyms, and foreign-language terms, all of which must be converted to fully pronounceable Vietnamese words before TTS synthesis or downstream language processing. Existing Vietnamese normalization tools either require heavy neural dependencies while covering only a narrow subset of NSW classes, or are embedded within larger NLP toolkits without standalone installability. VietNormalizer addresses these gaps through a unified, rule-based pipeline that: (1) converts arbitrary integers, decimals, and large numbers to Vietnamese words; (2) normalizes dates and times to their spoken Vietnamese forms; (3) handles VND and USD currency amounts; (4) expands percentages; (5) resolves acronyms via a customizable CSV dictionary; (6) transliterates non-Vietnamese loanwords and foreign terms to Vietnamese phonetic approximations; and (7) performs Unicode normalization and emoji/special-character removal. All regular expression patterns are pre-compiled at initialization, enabling high-throughput batch processing with minimal memory overhead and no GPU or external API dependency. The library is installable via pip install vietnormalizer, available on PyPI and GitHub at https://github.com/nghimestudio/vietnormalizer, and released under the MIT license. We discuss the design decisions, limitations of existing approaches, and the generalizability of the rule-based normalization paradigm to other low-resource tonal and agglutinative languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。