用菲律宾本土文化设计的LLM评测套件，精准检验模型文化适配性
Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino
- 由菲律宾人共创150个精细文化提示，测试模型对本土知识的响应能力
- 最佳模型仅答对46.0%，远低于本地人89.1%的正确率，凸显文化差距
- 适合评估多语言大模型在菲律宾文化场景中的表现，尤其关注本土化
当前多语言大语言模型未必能为菲律宾用户提供文化恰当且相关的回应。我们提出Kalahi——一个由母语菲律宾人协作创建的文化评测套件。该套件包含150个高质量、手工设计且富含细微差别的提示，用于测试模型在共享菲律宾文化知识与价值观情境下的生成能力。模型在Kalahi中表现良好，意味着其输出与普通菲律宾人在特定情境下的言行相似。我们在具备多语言及菲律宾语支持的LLMs上进行了实验。结果显示，尽管对菲律宾人而言问题简单，但对模型而言却极具挑战：最佳模型仅正确回答46.0%的问题，而本地人平均正确率达89.10%。因此，Kalahi可准确、可靠地评估大模型在菲律宾文化表达上的表现。
原文摘要 · Abstract (English)
Multilingual large language models (LLMs) today may not necessarily provide culturally appropriate and relevant responses to its Filipino users. We introduce Kalahi, a cultural LLM evaluation suite collaboratively created by native Filipino speakers. It is composed of 150 high-quality, handcrafted and nuanced prompts that test LLMs for generations that are relevant to shared Filipino cultural knowledge and values. Strong LLM performance in Kalahi indicates a model's ability to generate responses similar to what an average Filipino would say or do in a given situation. We conducted experiments on LLMs with multilingual and Filipino language support. Results show that Kalahi, while trivial for Filipinos, is challenging for LLMs, with the best model answering only 46.0% of the questions correctly compared to native Filipino performance of 89.10%. Thus, Kalahi can be used to accurately and reliably evaluate Filipino cultural representation in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。