提出新提示方法，评估多语言大模型的混语翻译能力
Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities
- 设计规则提示法生成多样化混语句子
- 在5组语言对上测试3个模型，发现0/1/10/20次提示效果各异
- 构建标准混语数据集，支持真实场景聊天机器人应用
多语言大模型在机器翻译任务中表现优异，但在混语（同一语句中混合两种及以上语言）场景下的翻译能力仍待探索。本文提出规则提示法，用于生成混语句子，评估GPT-3.5-turbo、GPT-4和Gemini Pro在英语-印地语、英语-孟加拉语、英语-古吉拉特语、英语-法语、英语-西班牙语共五组语言对上的混语翻译能力。采用k-shot提示（k∈{0, 1, 10, 20}）进行测试，结果表明，虽然k-shot提示通常表现最佳，但规则提示能生成风格多样的混语表达。同时构建了涵盖五组语言对的高质量混语数据集，用于评估模型将混语翻译为英文的能力，并基于该成果开发了混语聊天机器人。
原文摘要 · Abstract (English)
Multilingual Large Language Models (LLMs) have demonstrated exceptional performance in Machine Translation (MT) tasks. However, their MT abilities in the context of code-switching (the practice of mixing two or more languages in an utterance) remain under-explored. In this paper, we introduce Rule-Based Prompting, a novel prompting technique to generate code-mixed sentences. We measure and compare the code-mixed MT abilities of 3 popular multilingual LLMs: GPT-3.5-turbo, GPT-4, and Gemini Pro across five language pairs: English-{Hindi, Bengali, Gujarati, French, Spanish} using $k$-shot prompting ($k\in\{0, 1, 10, 20\}$) and Rule-Based Prompting. Our findings suggest that though $k$-shot prompting often leads to the best results, Rule-Based prompting shows promise in generating unique code-mixed sentences that vary in their style of code-mixing. We also use $k$-shot prompting to gauge the code-mixed to English translation abilities of multilingual LLMs. For this purpose, we create a gold-standard code-mixed dataset spanning five language pairs: English-{Hindi, Bengali, Gujarati, French, Spanish}. As a real-world application of our work, we create a code-mixed chatbot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。