Evaluating selected AI tools for literature reviews: a case study in local administration cybersecurity

Purpose: The objectives of this article are to identify scientific articles on the cybersecurity of local administration and to assess how useful the aid of two Artificial Intelligence tools: SciSpace.com (Advanced Plan) and Consensus.app (Pro tier) can be in this type of research. Design/methodology/approach: The method of conducting the screening and verifying its results based on manual screening partially assisted by LLMs is described in detail in the article. The PRISMA 2020 protocol was used for a part of the preliminary screening. Findings: The study indicates that both tools have limited usefulness for screening based on abstract content and better for full text document screening. When using the Scopus database, the study showed minimal utility of the PRISMA protocol implementation within the SciSpace tool, particularly when weighed against its usage cost. Research limitations/implications: Due to both: the nature of LLM tools, whose immanent feature is indeterminism, often leading to the lack of reproducibility of obtained results, and the business model, where the tools are continually developed and their behavior changes over time, the obtained results are not general or universal. Practical implications: Local administration cybersecurity is a very specific field of knowledge and practice. This type of literature review can be helpful to those working in said field in different countries. and the identification of the tested AI tools’ limitations – particularly the relatively low quality of screening based solely on article abstracts – may allow for more effective use of similar tools. Originality/value: The results provide knowledge about publications in the field of cybersecurity in local government administration and include calculations of the basic statistics (accuracy, sensitivity, and specificity) of the discussed tools in the examined case. To our knowledge, this is the most current and comprehensive literature review on this topic to date, while the calculations constitute a contributory study. Keywords: information security management systems, local administration cybersecurity. Category of the paper: research paper.