用大模型做文本预处理，效果比传统方法更好。
Investigating Large Language Models' Linguistic Abilities for Text Preprocessing
- 用大模型结合上下文进行停用词、词干化和词形还原
- 在六种欧洲语言上准确率分别达97%、82%、74%
- 经大模型预处理的文本让分类模型F1提升最高6%
文本预处理是自然语言处理的基础环节，包括去除停用词、词干化和词形还原等技术，用于为后续处理准备文本数据。尽管这些技术具有明显的上下文依赖性，但传统方法通常忽略上下文信息。本文探讨使用大型语言模型（LLMs）执行多种预处理任务的可行性，因其能无需大量语言特定标注资源即可考虑上下文。我们在网络获取的数据上进行了全面评估，对比了基于LLM的预处理（特别是停用词移除、词形还原和词干化）与传统算法在六个欧洲语言的多个文本分类任务中的表现。分析表明，LLMs在复制传统停用词移除、词形还原和词干化方法方面表现良好，准确率分别达到97%、82%和74%。此外，使用LLM预处理后的文本训练的机器学习模型，在F1指标上相比传统方法最高可提升6%。代码、提示和结果已公开于https://github.com/GianCarloMilanese/llm_pipeline_wi-iat。
原文摘要 · Abstract (English)
Text preprocessing is a fundamental component of Natural Language Processing, involving techniques such as stopword removal, stemming, and lemmatization to prepare text as input for further processing and analysis. Despite the context-dependent nature of the above techniques, traditional methods usually ignore contextual information. In this paper, we investigate the idea of using Large Language Models (LLMs) to perform various preprocessing tasks, due to their ability to take context into account without requiring extensive language-specific annotated resources. Through a comprehensive evaluation on web-sourced data, we compare LLM-based preprocessing (specifically stopword removal, lemmatization and stemming) to traditional algorithms across multiple text classification tasks in six European languages. Our analysis indicates that LLMs are capable of replicating traditional stopword removal, lemmatization, and stemming methods with accuracies reaching 97%, 82%, and 74%, respectively. Additionally, we show that ML algorithms trained on texts preprocessed by LLMs achieve an improvement of up to 6% with respect to the $F_1$ measure compared to traditional techniques. Our code, prompts, and results are publicly available at https://github.com/GianCarloMilanese/llm_pipeline_wi-iat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。