用哈夫曼编码优化非拉丁语种的音译,提升低资源语言模型表现。
Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languages
- 将音译与哈夫曼编码结合,实现无损压缩和高效处理。
- 文件大小减少50%,词元数降低50%-80%,保持原文完全可恢复。
- 适合需要处理低资源语言的NLP研究者和工程师使用。
随着大语言模型(LLMs)在日益多样和庞大的多语言语料上训练,其跨语言迁移能力逐渐显现。然而,这些能力难以有效延伸至低资源语言,尤其是使用非拉丁字母的语言。尽管将低资源语言音译为拉丁字母是自然解决方案,但目前缺乏将其融入模型训练与部署的完整框架。本文提出一种创新方法:将字符音译与哈夫曼编码相结合,构建完整的音译框架。该框架具备四大优势:1)压缩性:显著降低存储需求,文件大小减少达50%,词元数量减少50%-80%;2)准确性:确保音译文本可100%无损还原为源语言;3)效率:无需为低资源语言扩展词表,提升训练与推理效率;4)可扩展性:适用于其他低资源语言。我们在文本分类、机器阅读理解及机器翻译等多个下游任务中验证了该框架的有效性。实验表明,该方法显著增强了模型对低资源语言的处理能力,同时保持对高资源语言的性能。相关数据与代码已公开于 https://github.com/CMLI-NLP/HuffmanTranslit。
原文摘要 · Abstract (English)
As large language models (LLMs) are trained on increasingly diverse and extensive multilingual corpora, they demonstrate cross-lingual transfer capabilities. However, these capabilities often fail to effectively extend to low-resource languages, particularly those utilizing non-Latin scripts. While transliterating low-resource languages into Latin script presents a natural solution, there currently lacks a comprehensive framework for integrating transliteration into LLMs training and deployment. Taking a pragmatic approach, this paper innovatively combines character transliteration with Huffman coding to design a complete transliteration framework. Our proposed framework offers the following advantages: 1) Compression: Reduces storage requirements for low-resource language content, achieving up to 50% reduction in file size and 50-80% reduction in token count. 2) Accuracy: Guarantees 100% lossless conversion from transliterated text back to the source language. 3) Efficiency: Eliminates the need for vocabulary expansion for low-resource languages, improving training and inference efficiency. 4) Scalability: The framework can be extended to other low-resource languages. We validate the effectiveness of our framework across multiple downstream tasks, including text classification, machine reading comprehension, and machine translation. Experimental results demonstrate that our method significantly enhances the model's capability to process low-resource languages while maintaining performance on high-resource languages. Our data and code are publicly available at https://github.com/CMLI-NLP/HuffmanTranslit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。