用字符二元组和词频分析实现高精度语言识别，无需依赖AI模型。
Language Detection by Means of the Minkowski Norm: Identification Through Character Bigrams and Frequency Analysis
- 基于字符二元组与词频排名的数学方法，不依赖深度学习。
- 短文本（<150字符）准确率超80%，长文本达100%。
- 适合对轻量级、可解释性语言检测有需求的研究者。
近年来，随着AI语言模型的快速发展，语言识别问题再度引发关注。然而，非AI方法却逐渐被忽视。本研究通过借鉴已有语言学研究成果，采用单字与双字频率排序的数学算法实现语言判定。实验使用涵盖不同长度、历史时期和体裁的文本数据集，包括短篇小说、童话和诗歌。尽管存在多样性，该方法在短文本（少于150字符）上仍保持超过80%的准确率，长文本则达到100%。结果表明，传统的频率分析方法在语言识别中依然有效且具备良好可扩展性，是替代人工智能模型的一种可行方案。
原文摘要 · Abstract (English)
The debate surrounding language identification has gained renewed attention in recent years, especially with the rapid evolution of AI-powered language models. However, the non-AI-based approaches to language identification have been overshadowed. This research explores a mathematical implementation of an algorithm for language determinism by leveraging monograms and bigrams frequency rankings derived from established linguistic research. The datasets used comprise texts varying in length, historical period, and genre, including short stories, fairy tales, and poems. Despite these variations, the method achieves over 80\% accuracy on texts shorter than 150 characters and reaches 100\% accuracy for longer texts. These results demonstrate that classical frequency-based approaches remain effective and scalable alternatives to AI-driven models for language detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。