提出无监督框架识别大模型生成文本中的多元观点，揭示其观点多样性不足的问题。
Evaluating Pluralism in LLMs through Latent Perspectives
- 构建跨领域无监督多层框架，自动提取文本中隐藏的观点
- 在书评数据集上发现大模型生成内容观点分布偏离人类水平，稀有观点仍被忽略
- 适用于评估大模型在观点多样性上的对齐程度，适合关注公平性与多样性研究者
随着对多元观点表达需求的增加，具有多样性的大语言模型生成受到关注。尽管难以操作化，但识别文本中体现的观点可为多元对齐提供明确指引，并更清晰地揭示大模型生成中的观点差距。现有研究表明，模型会减少训练数据的多样性并生成同质化内容，但大多基于选择题或自由文本的高层特征。本文提出并实现一种无需领域知识的多层无监督框架，用于识别大模型生成文本中的多元观点。我们在高度主观的书评数据集上评估该框架，对比不同提示和模型。结果表明，虽部分模型与提示接近覆盖广泛观点，但稀有观点仍显著缺失，导致生成内容分布偏离人类文本。
原文摘要 · Abstract (English)
The growing need to represent diverse perspectives has increased interest in pluralistic LLM generation. Although difficult to operationalize, identifying perspectives expressed in text would provide clear guidance on pluralistic alignment and more clearly articulate the pluralistic gap in LLM generation. While models have been shown to reduce the diversity of training data and generate homogeneously, this has been demonstrated primarily on multiple-choice questionnaires or using high-level characteristics of free-form text. In this paper, we introduce and implement a domain-agnostic multi-layered framework for unsupervised extraction of perspectives suitable for identifying the pluralistic gap in LLM-generated text. We evaluate our framework on book reviews, a highly opinionated dataset representing diverse perspectives, and compare various prompts and models. Our results show that while some models and prompting techniques come close to covering a broad spectrum of perspectives, rarer perspectives remain disproportionately underrepresented, resulting in distributions that diverge from human text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。