用DeepL翻译问答数据中的答案片段，轻松生成芬兰语SQuAD数据集。
Finnish SQuAD: A Simple Approach to Machine Translation of Span Annotations
- 用DeepL直接翻译带格式的标注数据，无需复杂处理。
- 芬兰语版SQuAD在下游模型上表现优于其他翻译方法。
- 适合低资源语言的问答数据构建，开源可用。
我们采用一种简单方法，利用DeepL机器翻译服务及其对格式化文档的翻译能力，将带有跨度标注的数据集进行机器翻译。通过该方法，我们生成了芬兰语版的SQuAD2.0问答数据集，并在此新数据集上训练问答检索模型。通过直接评估、与类似数据集的间接对比、反向翻译实验以及下游模型性能测试，全面评估了数据质量和翻译方法的有效性。所有评估均表明该方法不仅使用简便，且生成的翻译数据质量更优。鉴于其在SQuAD数据集上的良好表现，该方法有望推广至其他任务和语言的跨度标注数据集翻译。相关代码与数据已开源：数据位于HuggingFace TurkuNLP/squad_v2_fi，代码在GitHub TurkuNLP/squad2-fi，模型在HuggingFace TurkuNLP/bert-base-finnish-cased-squad2。
原文摘要 · Abstract (English)
We apply a simple method to machine translate datasets with span-level annotation using the DeepL MT service and its ability to translate formatted documents. Using this method, we produce a Finnish version of the SQuAD2.0 question answering dataset and train QA retriever models on this new dataset. We evaluate the quality of the dataset and more generally the MT method through direct evaluation, indirect comparison to other similar datasets, a backtranslation experiment, as well as through the performance of downstream trained QA models. In all these evaluations, we find that the method of transfer is not only simple to use but produces consistently better translated data. Given its good performance on the SQuAD dataset, it is likely the method can be used to translate other similar span-annotated datasets for other tasks and languages as well. All code and data is available under an open license: data at HuggingFace TurkuNLP/squad_v2_fi, code on GitHub TurkuNLP/squad2-fi, and model at HuggingFace TurkuNLP/bert-base-finnish-cased-squad2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。