用简单模型高效生成德语方言词典，效果优于大模型。
Resource-Lean Lexicon Induction for German Dialects
- 用随机森林基于字符相似性训练词典生成模型。
- 在方言检索中提升50.7%召回率，超越大模型性能。
- 适合资源匮乏的方言研究者快速构建词库。
自动构建高质量词典对语言资源建设至关重要，但低资源语言和方言面临标注者稀缺、拼写差异大及大语言模型表现差等挑战。我们实证表明，基于字符串相似性特征训练的随机森林模型在德语方言词典诱导任务中表现优异，优于大语言模型，支持跨方言迁移，且计算开销小。在双语词典诱导（BLI）任务中，其性能超过Mistral-123b，同时更轻量。在方言信息检索（IR）中，使用该词典进行查询扩展，使nDCG@10提升最多达28.9%，Recall@100提升50.7%。为应对方言资源稀缺问题，我们进一步探究模型在不同德语方言间的迁移能力及其在不同训练数据量下的表现。
原文摘要 · Abstract (English)
Automatic induction of high-quality dictionaries is essential for building lexical resources, yet low-resource languages and dialects pose several challenges: limited access to annotators, high degree of spelling variations, and poor performance of large language models (LLMs). We empirically show that statistical models (random forests) trained on string similarity features are surprisingly effective for inducing German dialect lexicons. They outperform LLMs, enable cross-dialect transfer, and offer a lightweight data-driven alternative. We evaluate our models intrinsically on bilingual lexicon induction (BLI) and extrinsically on dialect information retrieval (IR). On BLI, random forests outperform Mistral-123b while being more resource-lean. On dialect IR with BM25, using our dialect dictionaries for query expansion yields relative improvements of up to 28.9% in nDCG@10 and 50.7% in Recall@100. Motivated by the resource scarcity in dialects, we further investigate the extent to which models transfer across different German dialects, and their performance under varying amounts of training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。