探查LLaMA模型中情感信息的隐藏位置，提升分析效率。
LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing
- 通过探测器分析各层情感特征，定位中层为情感信号集中区。
- 情感识别准确率比提示法高14%，内存需求平均降低57%。
- 适合想优化情感分析、降低推理开销的研究者与开发者。
大型语言模型（LLMs）在自然语言处理中迅速成为核心，可通过提示技术适应多种任务，包括情感分析。然而，我们对这些模型如何捕捉情感信息仍了解有限。本研究通过探测手段分析Llama模型各隐藏层的情感表征，以确定情感特征最显著的位置及其对情感分析的影响。利用探测分类器，我们评估了不同层与池化方法下的情感编码效果，发现二元极性任务中情感信息主要集中于中层，检测准确率相比提示技术最高提升14%。此外，在仅解码器模型中，最后一刻符并非始终最具信息量。该方法使情感任务的内存需求平均减少57%。研究揭示了层特定探测在超越提示的情感任务中的有效性，有助于提升模型实用性并降低内存开销。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have rapidly become central to NLP, demonstrating their ability to adapt to various tasks through prompting techniques, including sentiment analysis. However, we still have a limited understanding of how these models capture sentiment-related information. This study probes the hidden layers of Llama models to pinpoint where sentiment features are most represented and to assess how this affects sentiment analysis. Using probe classifiers, we analyze sentiment encoding across layers and scales, identifying the layers and pooling methods that best capture sentiment signals. Our results show that sentiment information is most concentrated in mid-layers for binary polarity tasks, with detection accuracy increasing up to 14% over prompting techniques. Additionally, we find that in decoder-only models, the last token is not consistently the most informative for sentiment encoding. Finally, this approach enables sentiment tasks to be performed with memory requirements reduced by an average of 57%. These insights contribute to a broader understanding of sentiment in LLMs, suggesting layer-specific probing as an effective approach for sentiment tasks beyond prompting, with potential to enhance model utility and reduce memory requirements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。