Topic counts and quality in topic models for historic corpora

Topic modelling methods enable the identification of potential topics within a corpus of historical texts, in particular they enable the identification of latent topics that are not described just by a single word. Like so many computational methods for the automatic processing of historical text corpora, they come with a number of parameters with which the method can be tuned and adapted. Each change in the settings of any of these parameters will generate a new set of topics that will differ in larger or smaller ways and which may be qualitatively better or worse. One of the main parameters for tuning topic models is setting the number of topics to be generated. In this paper we present an analysis of the impact of the number of topics on the quality of topic models for two historical text corpora. Two manual evaluation approaches are combined with an automated evaluation metric and based on the results we propose a formalised process for choosing the final set of parameters for a topic model. The process ensures the quality of the final model, while minimising the amount of manual evaluation work. The more structured process also allows for better documentation of the choices and in that way enables better reproducibility of any research using topic models.