arXiv:2505.07871cs.CLcs.AI2025-05被引 2

用标注者指令增强提示,让大模型更准理解金融文本情绪,提升股市预测效果。

Evaluating Financial Sentiment Analysis with Annotators Instruction Assisted Prompting: Enhancing Contextual Interpretation and Stock Prediction Accuracy

  • 将人工标注时的详细任务说明融入大模型提示,统一语义理解标准
  • 在WSB数据集上使大模型性能提升最高达9.08点,显著改善情绪识别
  • 创新性使用模型置信度构建情感指数,助力股票价格预测

金融情感分析(FSA)因金融语境中语言细微差异,对大模型提出更高挑战。现有基准数据集如Financial Phrasebank存在未明确定义的情感类别,导致标注主观性强、差异大,使大模型在评估中面临不合理期待——需揣测无充分上下文支持的人类主观观点。本文提出注释者指令辅助提示(AIAP),将原用于人类标注者的详细任务说明融入大模型提示框架,重新定义FSA任务,实现人机对情感理解的一致性,提供公平且上下文丰富的评估基础。基于来自WallStreetBets(WSB)的全新数据集验证,AIAP显著提升大模型性能,最高提升达9.08。该上下文感知方法还引入基于模型置信度的新情感索引方法,增强股票价格预测模型效果,凸显WSB作为金融文本关键来源的价值。本研究为改进FSA评估方法提供了新思路。

原文摘要 · Abstract (English)

Financial sentiment analysis (FSA) presents unique challenges to LLMs that surpass those in typical sentiment analysis due to the nuanced language used in financial contexts. The prowess of these models is often undermined by the inherent subjectivity of sentiment classifications in existing benchmark datasets like Financial Phrasebank. These datasets typically feature undefined sentiment classes that reflect the highly individualized perspectives of annotators, leading to significant variability in annotations. This variability results in an unfair expectation for LLMs during benchmarking, where they are tasked to conjecture the subjective viewpoints of human annotators without sufficient context. In this paper, we introduce the Annotators' Instruction Assisted Prompt, a novel evaluation prompt designed to redefine the task definition of FSA for LLMs. By integrating detailed task instructions originally intended for human annotators into the LLMs' prompt framework, AIAP aims to standardize the understanding of sentiment across both human and machine interpretations, providing a fair and context-rich foundation for sentiment analysis. We utilize a new dataset, WSBS, derived from the WallStreetBets subreddit to demonstrate how AIAP significantly enhances LLM performance by aligning machine operations with the refined task definitions. Experimental results demonstrate that AIAP enhances LLM performance significantly, with improvements up to 9.08. This context-aware approach not only yields incremental gains in performance but also introduces an innovative sentiment-indexing method utilizing model confidence scores. This method enhances stock price prediction models and extracts more value from the financial sentiment analysis, underscoring the significance of WSB as a critical source of financial text. Our research offers insights into both improving FSA through better evaluation methods.

金融情感分析大模型评估股票预测提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。