研究情绪分析中语气细微差异对大模型的影响,发现改写文本可提升准确率。
On the Impact of Language Nuances on Sentiment Analysis with Large Language Models: Paraphrasing, Sarcasm, and Emojis
- 通过改写和对抗性增强提升模型对讽刺、表情符号等语义细微差别的理解。
- 在核电相关数据上训练的模型面对讽刺时准确率仅30%,去除讽刺后提升至51%。
- 通用语料训练的模型对讽刺识别准确率达60%,适合需处理复杂语境的任务者。
大型语言模型(LLMs)在情感分析等任务中表现优异,但社交媒体数据质量对其准确性影响显著。本研究探讨了表情符号、讽刺等文本细微差别对情感分析的影响,尤其关注通过文本改写提升数据质量。为弥补讽刺标注数据不足,作者构建了一个包含5929条推文的人工标注数据集,用于评估模型在多种讽刺情境下的表现。结果表明,使用核电等主题特定数据集微调的模型,在存在讽刺时无法正确理解情感,准确率仅为30%,需通过外部干预如去除讽刺来提升性能;去除讽刺后准确率最高提升21%。而基于广泛话题的通用推文数据集训练的模型,在识别讽刺情绪方面表现更好,准确率达60%。此外,采用对抗性文本增强技术,通过对原文做微小改动生成合成样本,使模型对讽刺推文的鲁棒性和准确率提升至约85%。同时,对语言碎片化推文进行改写,将约40%低置信度标签的样本转为高置信度,整体情感分析准确率提高6%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated impressive performance across various tasks, including sentiment analysis. However, data quality--particularly when sourced from social media--can significantly impact their accuracy. This research explores how textual nuances, including emojis and sarcasm, affect sentiment analysis, with a particular focus on improving data quality through text paraphrasing techniques. To address the lack of labeled sarcasm data, the authors created a human-labeled dataset of 5929 tweets that enabled the assessment of LLM in various sarcasm contexts. The results show that when topic-specific datasets, such as those related to nuclear power, are used to finetune LLMs these models are not able to comprehend accurate sentiment in presence of sarcasm due to less diverse text, requiring external interventions like sarcasm removal to boost model accuracy. Sarcasm removal led to up to 21% improvement in sentiment accuracy, as LLMs trained on nuclear power-related content struggled with sarcastic tweets, achieving only 30% accuracy. In contrast, LLMs trained on general tweet datasets, covering a broader range of topics, showed considerable improvements in predicting sentiment for sarcastic tweets (60% accuracy), indicating that incorporating general text data can enhance sarcasm detection. The study also utilized adversarial text augmentation, showing that creating synthetic text variants by making minor changes significantly increased model robustness and accuracy for sarcastic tweets (approximately 85%). Additionally, text paraphrasing of tweets with fragmented language transformed around 40% of the tweets with low-confidence labels into high-confidence ones, improving LLMs sentiment analysis accuracy by 6%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。