为韩语金融文本构建专用评估基准，揭示低资源领域嵌入模型的潜力与局限
TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Model Bring? -- A Case Study on Korea Financial Texts
- 针对韩语金融领域设计专用评估集KorFinMTEB，融合语言与文化特性
- 在翻译版基准上表现良好，但在新基准上出现显著性能下降
- 强调低资源领域需定制化评估框架，适合研究多语言、小语种NLP的学者
嵌入模型的领域特异性对性能至关重要。然而，现有基准（如FinMTEB）主要面向高资源语言，导致低资源语言（如韩语）研究不足。直接翻译英文基准难以捕捉低资源领域的语言与文化差异。本文提出KorFinMTEB，一个专为韩语金融领域设计的新基准，充分反映其独特的文化特征。实验表明，尽管模型在翻译版FinMTEB上表现稳健，但在KorFinMTEB上暴露出深层次语义理解任务中的关键性能差距，凸显直接翻译的局限性。这一发现强调了建立包含语言特异性与文化细微差别的评估基准的必要性，呼吁发展更精准评估低资源领域嵌入模型的专用框架。
原文摘要 · Abstract (English)
Domain specificity of embedding models is critical for effective performance. However, existing benchmarks, such as FinMTEB, are primarily designed for high-resource languages, leaving low-resource settings, such as Korean, under-explored. Directly translating established English benchmarks often fails to capture the linguistic and cultural nuances present in low-resource domains. In this paper, titled TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Models Bring? A Case Study on Korea Financial Texts, we introduce KorFinMTEB, a novel benchmark for the Korean financial domain, specifically tailored to reflect its unique cultural characteristics in low-resource languages. Our experimental results reveal that while the models perform robustly on a translated version of FinMTEB, their performance on KorFinMTEB uncovers subtle yet critical discrepancies, especially in tasks requiring deeper semantic understanding, that underscore the limitations of direct translation. This discrepancy highlights the necessity of benchmarks that incorporate language-specific idiosyncrasies and cultural nuances. The insights from our study advocate for the development of domain-specific evaluation frameworks that can more accurately assess and drive the progress of embedding models in low-resource settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。