小模型微调比大模型更适合低资源下的代码混杂文本讽刺检测
Small Wins Big: Comparing Large Language Models and Domain Fine-Tuned Models for Sarcasm Detection in Code-Mixed Hinglish Text
- 用少量生成数据微调DistilBERT，实现高效讽刺识别
- 微调后的小模型准确率达84%，高于所有大模型
- 适合资源有限但需精准识别讽刺的场景
在多语言与代码混杂环境下，讽刺检测因结构差异、非正式表达及低资源语言特性而极具挑战。本研究对比了四种大语言模型（Llama 3.1、Mistral、Gemma 3、Phi-4）与一个微调过的DistilBERT模型在代码混杂的印地语-英语混合文本（Hinglish）中的讽刺检测表现。结果表明，使用极少由大模型生成的代码混杂数据进行顺序微调的DistilBERT模型，在零样本和少样本设置下均达到最高整体准确率84%，显著优于所有大模型。研究显示，在低资源与数据稀缺场景中，对小型Transformer模型进行领域自适应微调，可显著提升讽刺检测效果。
原文摘要 · Abstract (English)
Sarcasm detection in multilingual and code-mixed environments remains a challenging task for natural language processing models due to structural variations, informal expressions, and low-resource linguistic availability. This study compares four large language models, Llama 3.1, Mistral, Gemma 3, and Phi-4, with a fine-tuned DistilBERT model for sarcasm detection in code-mixed Hinglish text. The results indicate that the smaller, sequentially fine-tuned DistilBERT model achieved the highest overall accuracy of 84%, outperforming all of the LLMs in zero and few-shot set ups, using minimal LLM generated code-mixed data used for fine-tuning. These findings indicate that domain-adaptive fine-tuning of smaller transformer based models may significantly improve sarcasm detection over general LLM inference, in low-resource and data scarce settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。