对比孟加拉文学与新闻语料,发现文学文本更丰富多样且可读性更低。
Lexical and Statistical Analysis of Bangla Newspaper and Literature: A Corpus-Driven Study on Diversity, Readability, and NLP Adaptation
- 基于维卡斯帕提和印地科鲁普语料库,量化分析词汇多样性与结构复杂度。
- 文学语料在类型-词频比、双词多样性等指标上显著优于新闻语料。
- 融合文学数据可提升下游任务性能,适合构建更稳健的孟加拉语NLP系统。
本文通过大规模语料驱动方法,分析孟加拉语文学与新闻文本的词汇多样性、结构复杂度与可读性。研究采用当前最完整的文学语料(Vacaspati)和新闻语料(IndicCorp),考察类型-词频比(TTR)、异形词比率(HLR)、双词多样性、平均音节数与词长、以及对齐齐夫定律(Zipf's Law)的程度。结果显示,尽管规模较小,文学语料在双词多样性、HLR等指标上显著高于新闻语料;其困惑度更高,表明语言更不可预测。同时,文学语料更符合全球词频分布规律,熵值更高,冗余更低。可读性评估显示文学文本更复杂。将文学数据融入训练可提升模型在多种下游任务中的表现。
原文摘要 · Abstract (English)
In this paper, we present a comprehensive corpus-driven analysis of Bangla literary and newspaper texts to investigate their lexical diversity, structural complexity and readability. We undertook Vacaspati and IndicCorp, which are the most extensive literature and newspaper-only corpora for Bangla. We examine key linguistic properties, including the type-token ratio (TTR), hapax legomena ratio (HLR), Bigram diversity, average syllable and word lengths, and adherence to Zipfs Law, for both newspaper (IndicCorp) and literary corpora (Vacaspati).For all the features, such as Bigram Diversity and HLR, despite its smaller size, the literary corpus exhibits significantly higher lexical richness and structural variation. Additionally, we tried to understand the diversity of corpora by building n-gram models and measuring perplexity. Our findings reveal that literary corpora have higher perplexity than newspaper corpora, even for similar sentence sizes. This trend can also be observed for the English newspaper and literature corpus, indicating its generalizability. We also examined how the performance of models on downstream tasks is influenced by the inclusion of literary data alongside newspaper data. Our findings suggest that integrating literary data with newspapers improves the performance of models on various downstream tasks. We have also demonstrated that a literary corpus adheres more closely to global word distribution properties, such as Zipfs law, than a newspaper corpus or a merged corpus of both literary and newspaper texts. Literature corpora also have higher entropy and lower redundancy values compared to a newspaper corpus. We also further assess the readability using Flesch and Coleman-Liau indices, showing that literary texts are more complex.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。