arXiv:2607.26286cs.CL2026-07

测试不同提示范围和示例相似性对本地大模型翻译的影响

Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation

  • 对比单语言与多语言提示、随机/词法/嵌入相似示例的选择效果
  • 嵌入相似示例在强模型上表现最佳,但提升有限;小模型在多语言提示下出错率高
  • 强调评估大模型翻译需考虑提示设计和多目标一致性,不只看语言对和指标

大型语言模型(LLMs)正被广泛用于通用翻译任务,但其性能通常仅在单一提示格式下评估:将一个源句翻译成一种目标语言。实际使用中,用户可能要求单语翻译、多个相关语言同时翻译,或基于示例的条件翻译。本文研究提示范围与示例选择作为本地LLM机器翻译的实验变量。我们在九个欧盟官方语言的FLORES开发集全数据上,评估英语到罗曼语族和英语到日耳曼语族的翻译表现。比较了三个本地指令微调的LLM(llama3.2:3b、mistral:latest、qwen2.5:14b)与OPUS-MT和NLLB-200专用翻译基线。测试零样本提示和k=5少样本提示,采用随机、词法相似性、嵌入相似性三种示例选择策略,并对比单目标提示与请求整个语族语言的JSON格式多目标提示。结果表明,专用翻译系统整体仍最强,尤其在日耳曼语族;少样本提示对mistral:latest和qwen2.5:14b有帮助,但对llama3.2:3b有害;嵌入检索在强模型上平均最优,但优于随机和词法示例的效果不显著;多目标提示对强模型可行,但在小模型中暴露结构化输出缺陷。这些发现提示应不仅按语言对和指标评估大模型翻译,还需考虑提示范围、检索策略和多目标兼容性。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly used as general-purpose translation systems, but their behavior is usually evaluated under a single prompt shape: translate one source sentence into one target language. In practice, users may ask for one target language, for several related languages at once, or for translations conditioned on examples. This paper studies prompt scope and demonstration selection as experimental variables for local LLM machine translation. We evaluate English-to-Romance and English-to-Germanic translation on the full FLORES devtest split for nine official European Union languages. We compare three local instruction-tuned LLMs, llama3.2:3b, mistral:latest, and qwen2.5:14b, against dedicated MT baselines from OPUS-MT and NLLB-200. We test zero-shot prompting and k=5 few-shot prompting with random, lexical-similarity, and embedding-similarity demonstration selection. We also compare single-target prompts with JSON-formatted family-scope prompts that request all languages in a family at once. Results show that dedicated MT systems remain strongest overall, especially for Germanic languages. Few-shot prompting helps mistral:latest and qwen2.5:14b, but hurts llama3.2:3b; embedding retrieval is best on average for the stronger LLMs, but its advantage over random and lexical examples is modest. Family-scope prompting is feasible for stronger local LLMs but exposes structured-output failures in smaller models. These findings motivate evaluating LLM translation not only by language pair and metric, but also by prompt scope, retrieval strategy, and multi-target compliance.

机器翻译大模型评估提示工程多语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。