arXiv:2412.03242cs.CL2024-12被引 4

评测ChatGPT在中英俄时尚术语提取中的表现,发现其精度优于传统工具。

Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus

  • 用ChatGPT、SketchEngine和TBXTools对比提取时尚术语
  • ChatGPT在多术语场景下保持高精度,优于传统工具
  • 适合需跨语言术语定义的NLP研究者使用

本研究比较了SketchEngine、TBXTools和ChatGPT在英文-俄文时尚语料库上提取术语的准确性,并评估了ChatGPT生成的术语定义质量。研究基于从网络收集的可比时尚杂志语料库,构建了包含中英俄三语术语的黄金标准数据集,用于自动提取和定义评估。结果表明,尽管SketchEngine和TBXTools具备高召回率,但随着术语数量增加,其精度显著下降,整体性能受限。相反,ChatGPT在更多术语情况下维持甚至提升精度。对60个常用中英俄术语的定义分析显示,其跨语言定义保持合理准确性和一致性,但有时遗漏关键细节或引入冗余信息。研究揭示单一工具无法全面胜出,各类工具在术语提取与应用中各具优势。

原文摘要 · Abstract (English)

This paper compares the accuracy of the terms extracted using SketchEngine, TBXTools and ChatGPT. In addition, it evaluates the quality of the definitions produced by ChatGPT for these terms. The research is carried out on a comparable corpus of fashion magazines written in English and Russian collected from the web. A gold standard for the fashion terminology was also developed by identifying web pages that can be harvested automatically and contain definitions of terms from the fashion domain in English and Russian. This gold standard was used to evaluate the quality of the extracted terms and of the definitions produced. Our evaluation shows that TBXTools and SketchEngine, while capable of high recall, suffer from reduced precision as the number of terms increases, which affects their overall performance. Conversely, ChatGPT demonstrates superior performance, maintaining or improving precision as more terms are considered. Analysis of the definitions produced by ChatGPT for 60 commonly used terms in English and Russian shows that ChatGPT maintains a reasonable level of accuracy and fidelity across languages, but sometimes the definitions in both languages miss crucial specifics and include unnecessary deviations. Our research reveals that no single tool excels universally; each has strengths suited to particular aspects of terminology extraction and application.

术语提取ChatGPT跨语言NLP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。