arXiv:2410.03240cs.CL2024-10中稿 · COLING 2025被引 6

YouTube字幕可替代电影字幕,精准反映真实语言使用频率。

Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary?

  • 用清洗后的YouTube字幕提取词汇频率,方法简单高效。
  • 在5种语言中构建词频规范,与心理语言学数据高度相关。
  • 适合研究者用于低资源语言或需真实语料的场景。

词频是心理语言学中的关键变量,可用于建模人类对词汇的熟悉度,即使在大语言模型时代依然有效。电影字幕中的词频已被证明是日常语言暴露的良好近似。然而,许多语言的电影字幕难以获取,或主要源自英语翻译。本文表明,经过仔细处理的YouTube字幕所提取的词频,其表现可媲美甚至优于现有最佳资源。这些数据对缺乏高质量字幕或语音语料的语言尤为有用。我们利用YouTube字幕为中文、英文、印尼语、日语和西班牙语构建了词频规范,并评估其与词汇决策时间、词汇熟悉度及词汇复杂度的相关性。除了与两项心理语言学变量强相关外,仅用线性回归基于新词频的数据,在英语和日语的词汇复杂度预测任务中取得了新高分,超越了基于电影字幕训练的模型以及GPT-4。代码、词频列表、fastText词向量和统计语言模型已开源:https://github.com/naist-nlp/tubelex。

原文摘要 · Abstract (English)

Word frequency is a key variable in psycholinguistics, useful for modeling human familiarity with words even in the era of large language models (LLMs). Frequency in film subtitles has proved to be a particularly good approximation of everyday language exposure. For many languages, however, film subtitles are not easily available, or are overwhelmingly translated from English. We demonstrate that frequencies extracted from carefully processed YouTube subtitles provide an approximation comparable to, and often better than, the best currently available resources. Moreover, they are available for languages for which a high-quality subtitle or speech corpus does not exist. We use YouTube subtitles to construct frequency norms for five diverse languages, Chinese, English, Indonesian, Japanese, and Spanish, and evaluate their correlation with lexical decision time, word familiarity, and lexical complexity. In addition to being strongly correlated with two psycholinguistic variables, a simple linear regression on the new frequencies achieves a new high score on a lexical complexity prediction task in English and Japanese, surpassing both models trained on film subtitle frequencies and the LLM GPT-4. Our code, the frequency lists, fastText word embeddings, and statistical language models are freely available at https://github.com/naist-nlp/tubelex.

词频分析YouTube数据心理语言学多语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。