研究超参数如何影响大模型讽刺检测能力，发现模型大小和版本更新显著影响效果。
Assessing how hyperparameters impact Large Language Models' sarcasm detection performance
- 对比不同尺寸与版本的大模型，测试超参数对讽刺识别的影响。
- 微调后Llama-2-13b达0.83准确率和F1分数，接近人类水平。
- 模型性能随版本迭代波动，需每次发布后重新评估。
讽刺检测对人类和机器均具挑战性。本文研究了OpenAI的GPT与Meta的Llama-2模型在讽刺识别任务中的表现，基于其强大的自然语言理解能力与广泛应用。在SARC2.0数据集的pol-bal子集上，评估了不同规模、版本及超参数配置下的微调与零样本模型。实验表明，同一模型家族内，微调性能随模型规模单调提升；超参数调整亦显著影响结果。全精度Llama-2-13b在微调下达到0.83的准确率与F1分数，接近平均人类表现；零样本设置中，某GPT-4模型实现0.70准确率与0.75 F1分数，优于先前方法。此外，模型性能可能随每次发布而上升或下降，提示需在新版本发布后重新评估性能。
原文摘要 · Abstract (English)
Sarcasm detection is challenging for both humans and machines. This work explores how model characteristics impact sarcasm detection in OpenAI's GPT, and Meta's Llama-2 models, given their strong natural language understanding, and popularity. We evaluate fine-tuned and zero-shot models across various sizes, releases, and hyperparameters. Experiments were conducted on the political and balanced (pol-bal) portion of the popular Self-Annotated Reddit Corpus (SARC2.0) sarcasm dataset. Fine-tuned performance improves monotonically with model size within a model family, while hyperparameter tuning also impacts performance. In the fine-tuning scenario, full precision Llama-2-13b achieves state-of-the-art accuracy and $F_1$-score, both measured at 0.83, comparable to average human performance. In the zero-shot setting, one GPT-4 model achieves competitive performance to prior attempts, yielding an accuracy of 0.70 and an $F_1$-score of 0.75. Furthermore, a model's performance may increase or decline with each release, highlighting the need to reassess performance after each release.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。