An Evaluation of Large Language Models Using Analytic Tradecraft Standards

Artificial intelligence (AI) is not a new phenomenon and has been in use in the U.S. Intelligence Community in various forms for years. Generally limited to single source intelligence processing, the advent of Large Language Models (LLM) to the public suggests the use of AI as an all-source analytic tool. Despite the natural language question and answer format of current LLM, AI as a provider of true all-source analysis is likely several generations of improvement away. However, AI LLM are being used to brainstorm ideas, wordsmith and edit text, and outline approaches. To identify the strength and limitations of current LLM and see where they could be of most benefit to human intelligence analysts, the authors evaluated the output of different LLM against current analytic tradecraft standards and compared those to human analysts.