首个评测大模型理解波斯礼节的基准,发现其文化理解严重不足。
We Politely Insist: Your LLM Must Learn the Persian Art of Taarof
- 构建12类社交场景的波斯礼节评测集,由母语者验证
- 主流模型在符合礼节时准确率比母语者低40-48%
- 提示用波斯语可提升表现,但西方礼貌标准不适用
大型语言模型难以适应特定文化沟通规范,限制其在全球场景中的应用。本文聚焦伊朗社会中的波斯礼节(taarof),这是一种强调谦逊、尊重和间接表达的复杂礼貌体系,而现有文化评估基准中尚无此内容。我们提出首个针对taarof理解的评测基准TaarofBench,包含450个角色扮演情景,覆盖12类常见社交互动,经母语者验证。对五种前沿LLM的评估显示,当礼节恰当使用时,模型准确率比母语者低40-48%。性能在不同话题间有差异,使用波斯语提示可提升表现,且存在性别偏差。此外,常规礼貌评分高的回复常违反taarof规范,表明西方礼貌框架存在局限。通过监督微调和直接偏好优化,模型在文化契合度上分别提升21.8%和42.3%。通过33名参与者(11名母语者、11名有渊源者、11名非伊朗人)的人类研究,建立了不同程度熟悉度下的基准。本工作为开发更具文化敏感性的大模型奠定基础,推动更自然的社会交互应用。
原文摘要 · Abstract (English)
Large language models (LLMs) struggle to navigate culturally specific communication norms, limiting their effectiveness in global contexts. We focus on Persian taarof, a social norm in Iranian interactions, which is a sophisticated system of ritual politeness that emphasizes deference, modesty, and indirectness, yet remains absent from existing cultural benchmarks. We introduce TaarofBench, the first benchmark for evaluating LLM understanding of taarof, comprising 450 role-play scenarios covering 12 common social interaction topics, validated by native speakers. Our evaluation of five frontier LLMs reveals substantial gaps in cultural competence, with accuracy rates 40-48% below native speakers when taarof is culturally appropriate. Performance varies between interaction topics, improves with Persian-language prompts, and exhibits gender-based asymmetries. We also show that responses rated "polite" by standard metrics often violate taarof norms, indicating the limitations of Western politeness frameworks. Through supervised fine-tuning and Direct Preference Optimization, we achieve 21.8% and 42.3% improvement in model alignment with cultural expectations. Our human study with 33 participants (11 native Persian, 11 heritage, and 11 non-Iranian speakers) forms baselines in varying degrees of familiarity with Persian norms. This work lays the foundation for developing diverse and culturally aware LLMs, enabling applications that better navigate complex social interactions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。