用人格特质模拟降低大模型相关性判断中的阈值偏倚
Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing
- 给大模型注入不同人格特质，观察其对判断偏倚的影响
- 高开放性与低神经质人格显著减少偏倚现象
- 适合改进大模型评估系统，尤其关注心理机制的应用
近期研究探索将大语言模型用于可扩展的相关性标注，但发现其易受启动效应影响，即先前的判断会影响后续结果。尽管心理学理论指出人格特质与此类偏差相关，但模拟人格在大模型中是否产生类似效应尚不明确。本文研究大模型中五大性格特征对相关性标注中启动效应的影响，基于TREC 2021和2022深度学习赛道数据集，使用多个大模型进行实验。结果显示，高开放性与低神经质人格配置能持续降低启动敏感性；且最有效的性格配置在不同模型和任务类型间存在差异。基于此，提出人格提示方法以缓解阈值启动效应，将心理学证据与大模型评估实践相连接。
原文摘要 · Abstract (English)
Recent research has explored LLMs as scalable tools for relevance labeling, but studies indicate they are susceptible to priming effects, where prior relevance judgments influence later ones. Although psychological theories link personality traits to such biases, it is unclear whether simulated personalities in LLMs exhibit similar effects. We investigate how Big Five personality profiles in LLMs influence priming in relevance labeling, using multiple LLMs on TREC 2021 and 2022 Deep Learning Track datasets. Our results show that certain profiles, such as High Openness and Low Neuroticism, consistently reduce priming susceptibility. Additionally, the most effective personality in mitigating priming may vary across models and task types. Based on these findings, we propose personality prompting as a method to mitigate threshold priming, connecting psychological evidence with LLM-based evaluation practices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。