arXiv:2607.15238cs.CL2026-07

用对数比几何建模字符频次,实现高效精准的语言识别

Language Identification via Compositional Data Analysis: A Linear-Time Classifier Based on Log-Ratio Geometry

  • 将字符/双字频次转为单纯形向量,经中心对数比变换映射到零和空间
  • 在六种语言上测试,长文本下准确率超95%,且处理时间线性增长
  • 适合资源受限场景,兼具可解释性与计算效率

语言识别通常采用神经网络或基于统计的n-gram模型。神经方法需大量计算资源,而传统频次法虽具线性时间效率,但其距离度量不适用于组合数据。本文将字符与双字频次分布建模为单纯形上的组合向量,通过中心对数比(CLR)变换双射映射至ℝᴰ的(D−1)维零和子空间,使欧氏距离对应艾奇森距离。提出结合CLR变换的单字与双字特征及拉普拉斯平滑的流程,以应对稀疏性问题。在六种语言上进行评估,结果表明该方法在不同文本长度下均表现稳健,尤其在长序列中性能优异。研究显示,组合表示为语言识别提供了确定性且高效的替代方案,特别适用于需要可解释性与低资源消耗的场景。

原文摘要 · Abstract (English)

Language identification is commonly addressed using either neural architectures or statistical n-gram models. Neural approaches typically require substantial computational resources, whereas classical frequency-based methods offer efficient linear-time performance, but rely on distance metrics that are not always appropriate for compositional data. This work models character and bigram frequency distributions as compositional vectors constrained to the simplex and mapped via the centered log-ratio (CLR) transformation bijectively onto the $(D-1)$-dimensional zero-sum subspace of $\mathbb{R}^D$, where Euclidean distances correspond to Aitchison distances. A pipeline is proposed, combining CLR-transformed unigram and bigram features with Laplace smoothing to address sparsity. The method is evaluated on six languages. Experimental results show that the proposed approach achieves robust accuracy across different text lengths, with strong performance for longer sequences. These findings indicate that compositional representations provide a deterministic and computationally efficient alternative for language identification, particularly in settings where interpretability and low resource consumption are essential.

语言识别组合数据对数比高效算法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。