梳理了情感分析中预处理技术的最优顺序，提升模型效果。
Best Preprocessing Techniques for Sentiment Analysis
- 按顺序先分词、再清洗文本、然后归一化、最后去停用词
- 分词最重要，拼写纠正影响最小，去停用词不删否定词更优
- 为实践者提供无需试错的高效预处理方案
推特数据的情感分析对监测产品舆论及政治社会运动具有重要意义。预处理作为机器学习算法前的关键步骤，能有效降噪并提升效率。然而，现有研究很少系统考察预处理技术的执行顺序。本研究发现，在考虑顺序的情况下，拼写纠正影响最小，而分词最具影响力；词干提取与停用词删除可互换，但应保留否定词。最优顺序为：分词 → 文本清洗 → 词干提取 → 停用词删除。该结果为从业者提供了无需昂贵探索性实验即可提升模型性能的系统化方法。
原文摘要 · Abstract (English)
Sentiment analysis in Twitter datasets is important because it enables monitoring public opinion on products and analysis of political and social movements. One critical step is preprocessing: the automated processing of text for machine learning algorithms. Preprocessing plays a critical role in reducing noise and improving efficiency. However, little research has systematically examined the order in which preprocessing techniques are implemented. We find that, when accounting for order, spelling correction is the least impactful preprocessing technique, whereas tokenisation is the most impactful. Stemming and stop-word removal are interchangeable, and it is better to remove stop words without removing negation. The best order for applying the preprocessing techniques was tokenisation, text cleaning, stemming, and then stopword removal. Our results provide a systematic approach for practitioners to deploy preprocessing to improve model output without the costly preprocessing exploratory phase.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。