外部信息反而降低大模型立场检测准确率，暴露其倾向性偏差风险。
Is External Information Useful for Stance Detection with LLMs?
- 测试8个大模型在3个数据集上使用维基百科和网络搜索信息的性能表现
- 多数情况下宏F1下降最多达27.9%，因模型更信外部文本立场而非原文
- 链式思考提示无法缓解问题，微调可部分改善但不能根除偏差
立场检测任务中，需判断文本对目标持支持、反对或中立态度。以往研究表明，使用外部信息（如维基百科摘录）可提升性能。然而，大语言模型（LLMs）是否受益于此类信息仍不清楚。本研究系统评估了维基百科与网络搜索信息对8个大型语言模型在3个数据集共12个目标上的影响。结果显示，绝大多数情况下性能反而下降，宏F1最高降低27.9%。实验表明，模型倾向于采纳所提供信息的立场与情感，而非原文真实立场。该现象在链式思考提示下依然存在，微调虽能缓解但未能完全消除偏差。此结果与基于BERT系统的已有结论相反，揭示了基于大模型的立场分类器面临的信息偏见风险。
原文摘要 · Abstract (English)
In the stance detection task, a text is classified as either favorable, opposing, or neutral towards a target. Prior work suggests that the use of external information, e.g., excerpts from Wikipedia, improves stance detection performance. However, whether or not such information can benefit large language models (LLMs) remains an unanswered question, despite their wide adoption in many reasoning tasks. In this study, we conduct a systematic evaluation on how Wikipedia and web search external information can affect stance detection across eight LLMs and in three datasets with 12 targets. Surprisingly, we find that such information degrades performance in most cases, with macro F1 scores dropping by up to 27.9\%. We explain this through experiments showing LLMs' tendency to align their predictions with the stance and sentiment of the provided information rather than the ground truth stance of the given text. We also find that performance degradation persists with chain-of-thought prompting, while fine-tuning mitigates but does not fully eliminate it. Our findings, in contrast to previous literature on BERT-based systems which suggests that external information enhances performance, highlight the risks of information biases in LLM-based stance classifiers. Code is available at https://github.com/ngqm/acl2025-stance-detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。