构建更精准的文化对齐评估数据集，提升LLM文化适应性测试能力
Progressing beyond Art Masterpieces or Touristic Clichés: how to assess your LLMs for cultural alignment?
- 基于新设计准则构建文化对齐评估数据集
- 实验显示新数据集能更好区分文化专精与非专精模型
- 适合关注大模型文化适应性的研究者和开发者
尽管大型语言模型（LLMs）的文化对齐问题日益受到关注，常被视作文化偏见，但针对文化评估的数据集设计与开发仍相对不足。本文回顾现有数据集方法并识别其主要局限，提出面向标注者的全新设计准则，并据此构建了一个新数据集。进一步开展了一系列对比实验，结果表明，该设计显著提升了测试集的区分能力，能在其他条件相同的情况下有效区分专精于特定文化的模型与非专精模型。
原文摘要 · Abstract (English)
Although the cultural (mis)alignment of Large Language Models (LLMs) has attracted increasing attention -- often framed in terms of cultural bias -- until recently there has been limited work on the design and development of datasets for cultural assessment. Here, we review existing approaches to such datasets and identify their main limitations. To address these issues, we propose design guidelines for annotators and report on the construction of a dataset built according to these principles. We further present a series of contrastive experiments conducted with this dataset. The results demonstrate that our design yields test sets with greater discriminative power, effectively distinguishing between models specialized for a given culture and those that are not, ceteris paribus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。