用方言引导的LLM评估低资源翻译，提升无参考评分准确率
LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmark
- 基于方言引导提示和词汇增强，改进LLM翻译评估
- 在斯利赫蒂-英语数据集上实现最高0.1083的斯皮尔曼相关性提升
- 适合低资源语言翻译评估研究者使用
低资源语言机器翻译评估面临持续挑战，主要因高质量参考译文稀缺。该问题在多方言语言中尤为严重，语言多样性与数据匮乏导致评估困难。大语言模型（LLMs）通过无参考评估技术提供解决方案，但缺乏方言上下文时效果下降。本文提出一种方言引导的综合评估框架：扩展ONUBAD数据集，加入斯利赫蒂-英语句对、机器翻译结果及母语者标注的直接评估（DA）分数；为弥补词汇差距，扩充分词器词汇表；引入回归头实现标量得分预测，并设计方言引导（DG）提示策略。多LLM实验表明，所提方法在各项设置中均优于现有方法，最大斯皮尔曼相关性提升达+0.1083。代码与数据集已公开。
原文摘要 · Abstract (English)
Evaluating machine translation (MT) for low-resource languages poses a persistent challenge, primarily due to the limited availability of high quality reference translations. This issue is further exacerbated in languages with multiple dialects, where linguistic diversity and data scarcity hinder robust evaluation. Large Language Models (LLMs) present a promising solution through reference-free evaluation techniques; however, their effectiveness diminishes in the absence of dialect-specific context and tailored guidance. In this work, we propose a comprehensive framework that enhances LLM-based MT evaluation using a dialect guided approach. We extend the ONUBAD dataset by incorporating Sylheti-English sentence pairs, corresponding machine translations, and Direct Assessment (DA) scores annotated by native speakers. To address the vocabulary gap, we augment the tokenizer vocabulary with dialect-specific terms. We further introduce a regression head to enable scalar score prediction and design a dialect-guided (DG) prompting strategy. Our evaluation across multiple LLMs shows that the proposed pipeline consistently outperforms existing methods, achieving the highest gain of +0.1083 in Spearman correlation, along with improvements across other evaluation settings. The dataset and the code are available at https://github.com/180041123-Atiq/MTEonLowResourceLanguage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。