构建首个覆盖18语言的代码混用评测基准，揭示大模型在跨语系混用中的短板。
CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages
- 设计8类任务+18语言的综合评测框架，涵盖传统NLP与LLM专用任务。
- 发现大模型在跨语系代码混用任务中表现普遍不足，尤其在不同语族间差距显著。
- 提出基于GPT-4提示的合成方法，可生成大规模真实感代码混用数据。
代码混用（code-mixing）指对话中语言切换的现象，对传统自然语言处理带来独特挑战。现有评测局限于少数语言对和任务，难以全面评估大语言模型（LLMs）的代码混用能力。尽管多语言用户亟需此能力，相关研究仍较匮乏，且代码混用数据合成技术发展不足。为此，我们提出CodeMixBench，一个涵盖8个任务（3个专为LLM设计，5个为传统NLP任务）、18种语言（横跨7个语言家族）的综合性评测基准。同时，我们提出一种新方法：结合词替换与GPT-4提示生成大规模合成代码混用文本。评估显示，所有模型在涉及不同语言家族的代码混用数据上均表现不佳。增加训练数据量、扩大模型规模及采用少样本学习可提升性能。代码与数据集已开源至https://github.com/Jeromeyluck/CodeMixBench。
原文摘要 · Abstract (English)
Code-mixing, the practice of switching between languages within a conversation, poses unique challenges for traditional NLP. Existing benchmarks are limited by their narrow language pairs and tasks, failing to adequately assess large language models' (LLMs) code-mixing abilities. Despite the recognized importance of code-mixing for multilingual users, research on LLMs in this context remains sparse. Additionally, current techniques for synthesizing code-mixed data are underdeveloped to generate code-mixing. In response, we introduce CodeMixBench, a comprehensive benchmark covering eight tasks, including three specific to LLMs and five traditional NLP tasks, and 18 languages across seven language families. We also propose a new method for generating large-scale synthetic code-mixed texts by combining word substitution with GPT-4 prompting. Our evaluation reveals consistent underperformance of LLMs on code-mixed datasets involving different language families. Enhancements in training data size, model scale, and few-shot learning could improve their performance. The code and dataset are available at https://github.com/Jeromeyluck/CodeMixBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。