False alarms, real damage: Adversarial attacks using LLM-based models on text-based Cyber Threat Intelligence systems

Cyber Threat Intelligence (CTI) has emerged as a vital complementary approach that operates in the early phases of the cyber threat lifecycle. CTI involves collecting, processing, and analysing threat data to provide a more accurate and rapid understanding of cyber threats. Due to the large volume of data, automation through Machine Learning (ML) and Natural Language Processing (NLP) models is essential for effective CTI extraction. These automated systems leverage Open Source Intelligence (OSINT) from sources like social networks, forums, and blogs to identify Indicators of Compromise (IoCs). Although prior research has focused on adversarial attacks on specific ML models, this study expands the scope by investigating vulnerabilities within various components of the entire CTI pipeline and their susceptibility to adversarial attacks. It particularly focuses on a system-level analysis of how existing adversarial techniques interact with and across multiple stages of this pipeline, resulting in cascading attack effects. These vulnerabilities arise because they ingest textual inputs from various open sources, including real and potentially fake content. Three types of attacks against CTI pipelines are analysed – evasion, flooding, and poisoning – and their impact on the system’s information selection capabilities is assessed. Specifically, focusing on fake text generation, the work demonstrates how adversarial text generation techniques can create fake cybersecurity and cybersecurity-like text that misleads classifiers, degrades performance, and disrupts system functionality. The focus is primarily on the evasion attack, as it precedes and enables flooding and poisoning attacks within the CTI pipeline. The findings reveal that the False Positive Rate (FPR) for evasion attacks reached 97% with a specialised ML classifier model, indicating the model’s high vulnerability to adversarial samples. Additionally, an FPR of 75% is observed for ChatGPT-4o as a classifier, indicating its susceptibility to adversarial examples. These results underscore the need for an additional verification component at the early stage of the CTI pipeline to detect and filter out misinformation before it spreads through the system.