对比三种日文分词工具在情感分类中的表现，发现SentencePiece效果最佳。
An Experimental Evaluation of Japanese Tokenizers for Sentiment-Based Text Classification
- 用TF-IDF+逻辑回归评估三种分词器性能
- SentencePiece组合在准确率上最优
- 适合日文文本分类任务的分词工具选型
本研究考察了三种主流分词工具（MeCab、Sudachi、SentencePiece）在日文情感分类任务中的表现。采用词频-逆文档频率（TF-IDF）向量化方法，评估朴素贝叶斯和逻辑回归两类传统机器学习分类器。结果表明，Sudachi生成的分词与词典定义高度一致，而MeCab和SentencePiece处理速度更快。在分类性能方面，SentencePiece配合TF-IDF与逻辑回归的组合表现最佳。
原文摘要 · Abstract (English)
This study investigates the performance of three popular tokenization tools: MeCab, Sudachi, and SentencePiece, when applied as a preprocessing step for sentiment-based text classification of Japanese texts. Using Term Frequency-Inverse Document Frequency (TF-IDF) vectorization, we evaluate two traditional machine learning classifiers: Multinomial Naive Bayes and Logistic Regression. The results reveal that Sudachi produces tokens closely aligned with dictionary definitions, while MeCab and SentencePiece demonstrate faster processing speeds. The combination of SentencePiece, TF-IDF, and Logistic Regression outperforms the other alternatives in terms of classification performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。