用大模型探索无监督分词极限，发现参数越多效果越好
Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models
- 让大模型先理解再分词，测试其语义能力
- 参数多的模型在多语言上表现更优
- 提出新方法LLACA，结合大模型与自动机优势
词分割是自然语言处理的基础。基于“先理解，后分词”的理念，本文提出新框架，探索大语言模型（LLMs）在无监督词分割中的极限，并评估其语义理解能力。我们使用主流大模型在多种语言上执行词分割，以检验其“理解”水平。结果表明，大模型能通过简单提示将原始文本正确分割为词语。数据显示，参数量更大的模型在多语言任务中表现更佳。此外，本文提出一种新型无监督方法LLACA（大型语言模型启发的阿霍-科拉西克自动机），利用阿霍-科拉西克自动机的模式识别能力，结合预训练大模型的深层语义洞察，构建可根据上下文动态调整的n-gram模型，显著优于传统方法。代码已开源。
原文摘要 · Abstract (English)
Word segmentation stands as a cornerstone of Natural Language Processing (NLP). Based on the concept of "comprehend first, segment later", we propose a new framework to explore the limit of unsupervised word segmentation with Large Language Models (LLMs) and evaluate the semantic understanding capabilities of LLMs based on word segmentation. We employ current mainstream LLMs to perform word segmentation across multiple languages to assess LLMs' "comprehension". Our findings reveal that LLMs are capable of following simple prompts to segment raw text into words. There is a trend suggesting that models with more parameters tend to perform better on multiple languages. Additionally, we introduce a novel unsupervised method, termed LLACA ($\textbf{L}$arge $\textbf{L}$anguage Model-Inspired $\textbf{A}$ho-$\textbf{C}$orasick $\textbf{A}$utomaton). Leveraging the advanced pattern recognition capabilities of Aho-Corasick automata, LLACA innovatively combines these with the deep insights of well-pretrained LLMs. This approach not only enables the construction of a dynamic $n$-gram model that adjusts based on contextual information but also integrates the nuanced understanding of LLMs, offering significant improvements over traditional methods. Our source code is available at https://github.com/hkr04/LLACA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。