用歌词探测大模型文化偏见，发现模型默认倾向北美文化
Probing Cultural Signals in Large Language Models through Author Profiling
- 用歌词零样本推断歌手性别与族裔，无须微调
- 多数模型倾向预测北美族裔，DeepSeek-1.5B更倾向亚洲
- 提出两个公平性指标，揭示Ministral-8B偏见最强
大型语言模型（LLMs）在社会影响应用中日益普及，引发对其编码文化偏见的担忧。我们通过评估模型在零样本设置下从歌曲歌词中进行作者画像的能力来探测这些表征，推断歌手的性别和族裔，无需任务特定微调。在超过10,000首歌词上评估多个开源模型，发现模型表现出非平凡的画像性能，但存在系统性文化对齐：大多数模型默认倾向北美洲族裔，而DeepSeek-1.5B更强烈地对齐亚洲族裔。这一现象既体现在模型的预测分布中，也反映在生成推理过程的分析中。为量化差异，我们引入两个公平性度量——模态准确率偏差（MAD）和召回偏差（RD），结果显示Ministral-8B在族裔判断中表现出最强偏见，而Gemma-12B表现最均衡。代码已公开于GitHub，结果可在HuggingFace获取。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed in applications with societal impact, raising concerns about the cultural biases they encode. We probe these representations by evaluating whether LLMs can perform author profiling from song lyrics in a zero-shot setting, inferring singers' gender and ethnicity without task-specific fine-tuning. Across several open-source models evaluated on more than 10,000 lyrics, we find that LLMs achieve non-trivial profiling performance but demonstrate systematic cultural alignment: most models default toward North American ethnicity, while DeepSeek-1.5B aligns more strongly with Asian ethnicity. This finding emerges from both the models' prediction distributions and an analysis of their generated rationales. To quantify these disparities, we introduce two fairness metrics, Modality Accuracy Divergence (MAD) and Recall Divergence (RD), and show that Ministral-8B displays the strongest ethnicity bias among the evaluated models, whereas Gemma-12B shows the most balanced behavior. Our code is available on [GitHub](https://github.com/ValentinLafargue/CulturalProbingLLM) and results on [HuggingFace](https://huggingface.co/datasets/ValentinLAFARGUE/AuthorProfilingResults).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。