英語テキストに含まれる単語の出現頻度に付随する不定性の評価
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Zipf's law is one of the well-known universal characteristics of natural language. To evaluate the validity of Zipf's law quantitatively, it is necessary to evaluate the statistical errors of the word frequencies in a text (referred to as "true error"). In this study, true errors in English texts were examined using standard deviations of the word frequency distributions obtained from the Brown Corpus. It was found that the true error is significantly larger than the Poisson error. We performed fits of word frequencies as a function of the frequency rank using Zipf's law, assigning either the true or Poisson errors to the data points. We then applied the χ² and Kolmogorov-Smirnov tests to compare consistency between the data and Zipf's law. The results show that the error's size was underestimated when using Poisson errors. In addition, the overall shape of the fitted function was similar, regardless of the type of error used.
Fields of Study & Subjects
Natural Language Processing Techniques
Statistical Mechanics and Entropy
Complex Systems and Time Series Analysis
Publication details
- Journal / source
- Institutional Repositories DataBase (IRDB)
- Volume
- 30
- Pages
- 109–120
- Publication date
- 1 January 2026
- Language
- JA