用大模型预测人对文本的主观感知，发现它学的是个人习惯而非社会特征。
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions
- 在五个任务上微调大模型，让其根据社会属性预测标注差异。
- 模型性能提升但主要依赖标注者个体行为，非真实社会特征关联。
- 质疑当前用大模型模拟社会差异的有效性，适合研究标注偏见的人看。
人们在主观问题上的标注存在自然差异，部分差异可能与社会人口学特征有关。尽管大语言模型（LLMs）已被用于数据标注，但近期研究表明，当提示包含社会人口学属性时，模型表现不佳，暗示其内在缺乏社会人口学知识。本文探究是否可通过训练使大模型成为准确的社会人口学标注者变异模型。基于一个包含五个任务、标准化社会人口学信息的精选数据集，我们发现模型在微调后确实在社会人口学提示下表现提升，但这种提升主要源于模型学习了标注者的个体行为模式，而非真正理解社会人口学特征与标注之间的联系。所有任务的结果均表明，模型与社会人口学特征之间几乎没有有意义的关联，这引发了对当前使用大模型模拟社会差异和行为的合理性质疑。
原文摘要 · Abstract (English)
People naturally vary in their annotations for subjective questions and some of this variation is thought to be due to the person's sociodemographic characteristics. LLMs have also been used to label data, but recent work has shown that models perform poorly when prompted with sociodemographic attributes, suggesting limited inherent sociodemographic knowledge. Here, we ask whether LLMs can be trained to be accurate sociodemographic models of annotator variation. Using a curated dataset of five tasks with standardized sociodemographics, we show that models do improve in sociodemographic prompting when trained but that this performance gain is largely due to models learning annotator-specific behaviour rather than sociodemographic patterns. Across all tasks, our results suggest that models learn little meaningful connection between sociodemographics and annotation, raising doubts about the current use of LLMs for simulating sociodemographic variation and behaviour.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。