利用先入为主效应优化大模型答题,不改模型只调选项顺序
Exploiting Primacy Effect To Improve Large Language Models
- 按问题语义重排答案选项,利用模型对开头选项的偏好
- 在多个MCQA数据集上提升准确率,最高增益达4.2%
- 适合做评测或部署时优化提示策略的研究者和工程师
大型语言模型在自然语言处理任务中表现优异,但与人类一样存在位置偏差,如先入为主效应(primacy effect),影响多选题作答准确性。本文发现,微调会加剧这一偏差,可能源于模型学习了人类的答题模式。为此,我们提出一种无需知晓正确答案的方法:根据答案选项与问题的语义相似度重新排序,优先放置最相关的选项。实验表明,该方法显著提升了多个MCQA数据集上的性能,平均准确率提升1.8%,最高达4.2%。研究揭示了偏差既是挑战也是可利用的资源,为设计更鲁棒的模型和优化提示策略提供了新思路。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become essential in many Natural Language Processing (NLP) tasks, leveraging extensive pre-training and fine-tuning to achieve high accuracy. However, like humans, LLMs exhibit biases, particularly positional biases such as primacy and recency effects, which can influence the accuracy of the answers. The primacy effect-where items presented first are more likely to be remembered or selected-plays a key role in Multiple Choice Question Answering (MCQA), where the order of answer options can affect prediction outcomes. This study focuses on primacy bias in fine-tuned LLMs: We first show that fine-tuning amplifies this bias, probably due to exposure to human-like patterns. Hence, we strategically leverage this effect by reordering response options based on semantic similarity to the query, without requiring knowledge of the correct answer. Our experimental results show that this approach significantly improves performance in MCQA. More generally, our findings underscore the dual nature of biases as both challenges and opportunities, offering insights for bias-aware model design and NLP applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。