用词频分布判断语言，小样本也能准识别。
What Language is This? Ask Your Tokenizer
- 基于词频概率模型，通过比较文本在各语言中的出现可能性来判断语言。
- 低资源下仅需每语言5个样本即达70%准确率，显著提升样本效率。
- 无需重训练即可加新语言，适合集成到现有语言模型流程中。
语言识别（LID）是多语言自然语言处理的关键环节，用于语料整理、训练数据分析和跨语言大模型评估。尽管在高资源语言上表现接近完美，现有系统在低资源和近似语言场景中仍不稳定。本文提出UniLID，一种基于UnigramLM分词算法的简单高效语言识别方法，利用其概率框架、参数估计与推理策略。核心思想是：给定一段文本，问它在哪个语言的词频分布下最可能出现？该方法数据与计算效率高，支持增量添加新语言而无需重新训练，可无缝集成至现有语言模型分词流程。实验表明，相比fastText、GlotLID和CLD3等主流基线，UniLID在标准基准上表现相当，低资源场景下样本效率大幅提升——每语言仅需5个标注样本即可达到约70%准确率，并在细粒度方言识别任务中取得显著优势。
原文摘要 · Abstract (English)
Language Identification (LID) is an important component of many multilingual natural language processing pipelines, where it facilitates corpus curation, training data analysis, and cross-lingual evaluation of large language models. Despite near-perfect performance on high-resource languages, existing systems remain brittle in low-resource and closely related language settings. We introduce UniLID, a simple and efficient LID method based on the UnigramLM tokenization algorithm, leveraging its probabilistic framing, parameter estimation technique and inference strategy. In short, to predict a string's language label, we simply ask: under which language's unigram distribution is this string most likely? Our formulation is data- and compute-efficient, supports incremental addition of new languages without retraining existing models, and can naturally be integrated into existing language model tokenization pipelines. Empirical evaluations against widely used baselines, including fastText, GlotLID and CLD3, show that UniLID achieves competitive performance on standard benchmarks, substantially improves sample efficiency in low-resource settings -- reaching ~70% accuracy with as few as five labeled samples per language -- and delivers large gains on fine-grained dialect identification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。