英語テキストに含まれる単語の出現頻度に付随する不定性の評価

Read full text Summarise with VIRAT
Authors: Y. Takubo, Aoi KUBOTA
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Zipf's law is one of the well-known universal characteristics of natural language. To evaluate the validity of Zipf's law quantitatively, it is necessary to evaluate the statistical errors of the word frequencies in a text (referred to as "true error"). In this study, true errors in English texts were examined using standard deviations of the word frequency distributions obtained from the Brown Corpus. It was found that the true error is significantly larger than the Poisson error. We performed fits of word frequencies as a function of the frequency rank using Zipf's law, assigning either the true or Poisson errors to the data points. We then applied the χ² and Kolmogorov-Smirnov tests to compare consistency between the data and Zipf's law. The results show that the error's size was underestimated when using Poisson errors. In addition, the overall shape of the fitted function was similar, regardless of the type of error used.

Fields of Study & Subjects

Natural Language Processing Techniques Statistical Mechanics and Entropy Complex Systems and Time Series Analysis

Publication details

Journal / source
Institutional Repositories DataBase (IRDB)
Volume
30
Pages
109–120
Publication date
1 January 2026
Language
JA