构建首个菲律宾语大模型评测基准,覆盖理解、推理与生成三类能力。
Batayan: A Filipino NLP benchmark for evaluating Large Language Models
- 基于本土语言专家校验,构建多任务菲律宾语数据集
- 发现主流大模型在菲律宾语上表现显著落后,凸显语言资源不足
- 适合关注低资源语言、文化适配NLP的研究者和开发者
近年来大型语言模型在高资源语言上展现出卓越能力,但对低资源语言的语义细微差别仍缺乏探索。本文提出Batayan,一个全面的菲律宾语基准,系统评估大模型在理解、推理和生成三大自然语言处理能力上的表现。该基准整合了八个任务,其中三个为首次针对菲律宾语构建,涵盖塔加洛语及混合语态(Taglish)语句。通过母语者驱动的严格改编与验证流程,确保数据在复杂形态与句法结构上的流畅性与真实性,有效缓解现有菲律宾语语料中的翻译腔偏见。我们报告了多种开源与商业大模型的实测结果,揭示出显著性能差距,反映出菲律宾语在预训练语料中代表性不足、建模其丰富形态与构句结构存在独特挑战,以及显式支持菲律宾语的重要性。此外,我们讨论了数据集构建中的实际困难,并提出建设文化与语言忠实资源的系统性解决方案。同时提供公开评测套件,为菲律宾语NLP的迭代发展提供社区协作基础。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have demonstrated remarkable capabilities on widely benchmarked high-resource languages. However, linguistic nuances of under-resourced languages remain unexplored. We introduce Batayan, a holistic Filipino benchmark that systematically evaluates LLMs across three key natural language processing (NLP) competencies: understanding, reasoning, and generation. Batayan consolidates eight tasks, three of which have not existed prior for Filipino corpora, covering both Tagalog and code-switched Taglish utterances. Our rigorous, native-speaker-driven adaptation and validation processes ensures fluency and authenticity to the complex morphological and syntactic structures of Filipino, alleviating the pervasive translationese bias in existing Filipino corpora. We report empirical results on a variety of open-source and commercial LLMs, highlighting significant performance gaps that signal the under-representation of Filipino in pre-training corpora, the unique hurdles in modeling Filipino's rich morphology and construction, and the importance of explicit Filipino language support. Moreover, we discuss the practical challenges encountered in dataset construction and propose principled solutions for building culturally and linguistically-faithful resources in under-represented languages. We also provide a public evaluation suite as a clear foundation for iterative, community-driven progress in Filipino NLP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。