为欧洲葡萄牙语打造专用翻译模型，解决资源匮乏语言变体的性能短板。
Tradutor: Building a Variety Specific Translation Model
- 构建专用于欧洲葡萄牙语的开源翻译模型
- 在两个基准数据集上超越现有开源系统，接近闭源领先水平
- 适合关注小语种、方言差异与语言公平的研究者
语言模型虽已广泛应用于各类系统，但对资源丰富语言如英语表现优异，却常忽略多地区语言中的方言和语体细微差别。例如欧洲葡萄牙语常被巴西葡萄牙语掩盖，导致各类语言任务表现不佳。为此，本文提出首个专为欧洲葡萄牙语设计的开源翻译模型，并构建了专门针对该任务的新数据集。自动评估结果显示，最佳模型在两个基准数据集上的表现优于现有开源翻译系统，接近工业级闭源系统的水平。通过公开数据集、模型与代码，旨在推动对代表性不足语言变体的研究，促进语言多样性表达的进展。
原文摘要 · Abstract (English)
Language models have become foundational to many widely used systems. However, these seemingly advantageous models are double-edged swords. While they excel in tasks related to resource-rich languages like English, they often lose the fine nuances of language forms, dialects, and varieties that are inherent to languages spoken in multiple regions of the world. Languages like European Portuguese are neglected in favor of their more popular counterpart, Brazilian Portuguese, leading to suboptimal performance in various linguistic tasks. To address this gap, we introduce the first open-source translation model specifically tailored for European Portuguese, along with a novel dataset specifically designed for this task. Results from automatic evaluations on two benchmark datasets demonstrate that our best model surpasses existing open-source translation systems for Portuguese and approaches the performance of industry-leading closed-source systems for European Portuguese. By making our dataset, models, and code publicly available, we aim to support and encourage further research, fostering advancements in the representation of underrepresented language varieties.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。