用大模型教小模型细粒度情感分析，效果更好还更省资源。
Distilling Fine-grained Sentiment Understanding from Large Language Models
- 用大模型生成情感解析内容，用来预训练小模型。
- 小模型在情感分析上提升6.00%的准确率，仅220万参数就超过Llama-2-7b。
- 小模型具备零样本分类能力，可媲美甚至超越老师模型。
细粒度情感分析（FSA）旨在从大量带情感文本中提取和总结用户观点。近期研究表明，大语言模型（LLMs）具备出色的语义理解能力。然而，直接部署LLMs进行FSA应用会带来高昂的推理成本。因此，本文研究将大模型中的细粒度情感理解能力迁移到小语言模型（SLMs）中。我们通过提示（prompt）让LLMs分析并解释给定评论的情感，然后利用生成的内容对SLMs进行预训练。此外，我们构建了一个全面的FSA评估基准，用于测试SLMs和LLMs的表现。在该基准上的大量实验表明：（1）知识蒸馏显著提升了SLMs在FSA任务上的性能，F1分数提升6.00%，且仅用220万参数的模型即可超越拥有70亿参数的Llama-2-7b；（2）蒸馏后的SLMs具备优秀的零样本情感分类能力，表现可匹配甚至超过其教师模型。结果表明，从大模型蒸馏细粒度情感理解是一种极具前景的方向。代码、数据及预训练模型权重将公开于https://github.com/HITSZ-HLT/FSA-Distillation。
原文摘要 · Abstract (English)
Fine-grained sentiment analysis (FSA) aims to extract and summarize user opinions from vast opinionated text. Recent studies demonstrate that large language models (LLMs) possess exceptional sentiment understanding capabilities. However, directly deploying LLMs for FSA applications incurs high inference costs. Therefore, this paper investigates the distillation of fine-grained sentiment understanding from LLMs into small language models (SLMs). We prompt LLMs to examine and interpret the sentiments of given reviews and then utilize the generated content to pretrain SLMs. Additionally, we develop a comprehensive FSA benchmark to evaluate both SLMs and LLMs. Extensive experiments on this benchmark reveal that: (1) distillation significantly enhances the performance of SLMs in FSA tasks, achieving a 6.00\% improvement in $F_1$-score, and the distilled model can outperform Llama-2-7b with only 220M parameters; (2) distillation equips SLMs with excellent zero-shot sentiment classification capabilities, enabling them to match or even exceed their teacher models. These results suggest that distillation from LLMs is a highly promising direction for FSA. We will release our code, data, and pretrained model weights at https://github.com/HITSZ-HLT/FSA-Distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。