Lexical and Statistical Analysis of Bangla Newspaper and Literature: A Corpus-Driven Study on Diversity, Readability, and NLP Adaptation

In this paper, we present a comprehensive corpus-driven analysis ofBangla literary and newspaper texts to investigate their lexical diversity,structural complexity and readability. We undertook Vācaspati andIndicCorp, which are the most extensive literature and newspaper-onlycorpora for Bangla. We examine key linguistic properties, including thetype-token ratio (TTR), hapax legomena ratio (HLR), Bigram diversity,average syllable and word lengths, and adherence to Zipf’s Law, for bothnewspaper (IndicCorp) and literary corpora (Vācaspati). Our findingsreveal that till ∼ 10 6 words, the newspaper corpora show a higher TTRthan the literary corpora; after that, the literary corpora have a higherTTR ratio. For all other features, such as Bigram Diversity and HLR,despite its smaller size, the literary corpus exhibits significantly higherlexical richness and structural variation. We have also demonstrated thata literary corpus adheres more closely to global word distribution proper-ties, such as Zipf’s law, than a newspaper corpus or a merged corpus ofboth literary and newspaper texts. Literature corpora also have higherentropy and lower redundancy values compared to a newspaper corpus.We also further assess the readability using Flesch and Coleman-Liau in-dices, showing that literary texts are more complex. Additionally, wetried to understand the diversity of corpora by building 𝑛-gram modelsand measuring perplexity. Our findings reveal that literary corpora havehigher perplexity than newspaper corpora, even for similar sentence sizes.This trend can also be observed for the English newspaper and literaturecorpus, indicating its generalizability. We also examined how the perfor-mance of models on downstream tasks is influenced by the inclusion ofliterary data alongside newspaper data. Our findings suggest that inte-grating literary data with newspapers improves the performance of modelson various downstream tasks.