Experimental Evaluation of AI-Augmented Cybersecurity Requirements Generation Leveraging LLMs’ Capabilities

When it comes to cybersecurity requirements, catalogs of standard controls are a double-edged sword: they offer reusable resources applicable anywhere, yet they entail tedious, error-prone manual instantiation work. As these cybersecurity controls are textual in nature, we analyze whether large language models (LLMs) may come to help automate that process. We ran an empirical evaluation where frontier versions of LLaMa, QWen, Mixtral, and GPT were asked, leveraging best prompting practice, to generate cybersecurity requirements for a realistic system, departing from its functional specification, plus definitions of abstract cybersecurity controls derived from ISO/IEC 27002. We compared the results to a ground truth created by human experts, and a gold standard pooled from the aggregation of all human- and LLM-produced results, considering phenomena such as hallucinations, new requirements discovered by LLMs, or partial overlaps. Results show that Open Weights LLMs can parallel human experts—albeit producing complementary sets of requirements. Relative F2 score (which prioritizes recall over precision) against the gold standard, reached 1.14 when compared in relation to human experts, achieved in the best configuration with a LLaMa 3.1 405B model, prompted with an explicit chain-of-thought pipeline made of three sequential steps and four parallel and coalesced runs, and moderate values for hyperparameters that balance diversity and determinism. Our work stands out for analyzing practical, applied capabilities of LLMs for security by design in non-code artifacts. Results show the feasibility of this approach, yet the recall values indicate that improvements are still to be expected as LLMs evolve.