测试大模型在简体与繁体中文下的偏见差异,发现响应倾向因任务而异。
Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese
- 设计两项真实场景任务:地区术语选择与地区姓名选择
- 多数模型在术语任务中偏爱简体,但在姓名任务中反而倾向繁体
- 揭示训练数据与分词差异可能导致语言偏见,适合关注AI公平性的研究者
尽管大语言模型(LLMs)在简体和繁体中文中的能力已有研究,但其在不同中文变体提示下的表现差异尚不明确。这种差异可能因忽略两岸文化背景而引发代表性伤害,并加剧教育、招聘等领域的决策风险。为此,我们设计两个基准任务:区域术语选择(要求模型根据描述选择两岸称呼不同的物品名称)和区域姓名选择(从包含简体与繁体姓名的列表中选择合适人选)。我们审计了11个主流商业及开源LLM服务的表现,涵盖主要基于英文、简体或繁体中文训练的模型。分析显示,模型偏见取决于任务与提示语言:多数模型在术语任务中更倾向于简体回答,但在姓名任务中却意外偏好繁体姓名。这些差异可能源于训练数据分布、书写字符偏好及分词方式的差异。本研究呼吁加强多语言变体下模型偏见的评估,并开源了基准数据集(https://github.com/brucelyu17/SC-TC-Bench),以支持未来可复现的评测。
原文摘要 · Abstract (English)
While the capabilities of Large Language Models (LLMs) have been studied in both Simplified and Traditional Chinese, it is yet unclear whether LLMs exhibit differential performance when prompted in these two variants of written Chinese. This understanding is critical, as disparities in the quality of LLM responses can perpetuate representational harms by ignoring the different cultural contexts underlying Simplified versus Traditional Chinese, and can exacerbate downstream harms in LLM-facilitated decision-making in domains such as education or hiring. To investigate potential LLM performance disparities, we design two benchmark tasks that reflect real-world scenarios: regional term choice (prompting the LLM to name a described item which is referred to differently in Mainland China and Taiwan), and regional name choice (prompting the LLM to choose who to hire from a list of names in both Simplified and Traditional Chinese). For both tasks, we audit the performance of 11 leading commercial LLM services and open-sourced models -- spanning those primarily trained on English, Simplified Chinese, or Traditional Chinese. Our analyses indicate that biases in LLM responses are dependent on both the task and prompting language: while most LLMs disproportionately favored Simplified Chinese responses in the regional term choice task, they surprisingly favored Traditional Chinese names in the regional name choice task. We find that these disparities may arise from differences in training data representation, written character preferences, and tokenization of Simplified and Traditional Chinese. These findings highlight the need for further analysis of LLM biases; as such, we provide an open-sourced benchmark dataset to foster reproducible evaluations of future LLM behavior across Chinese language variants (https://github.com/brucelyu17/SC-TC-Bench).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。