Failure Modes and Remediations in LLM-Driven Automated Machine Learning: A Case Study on Network Intrusion Detection

Read full text Summarise with VIRAT
Authors: Chuck Owen
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

I present an empirical study of an LLM-driven automated machinelearning (AutoML) loop applied to binary network intrusion detec-tion on the CICIDS-2017 dataset. The framework adapts Karpathy’sautoresearch concept to tabular classification: Claude Sonnet 4.6iteratively rewrites a detector script, which is executed as a sub-process and evaluated against held-out validation and test sets.Over four sessions and 27 runs, I document six distinct failuremodes: revert-loop stagnation, unsupervised feature engineeringcausing train/test distribution mismatch, resource budget viola-tions, platform encoding crashes, deprecated API incompatibilities,and cross-process pickle scope errors. Concrete remediations areprovided for each. My best model achieves weighted F1 = 0.9992 onCICIDS-2017, consistent with published benchmarks. The centralfinding is that near a performance ceiling, the primary challengeof LLM-driven AutoML is not model quality but system reliabil-ity: ensuring the agent explores meaningfully, respects executionconstraints, and produces evaluable outputs. I provide a reusable,dataset-agnostic framework and a reproducible failure taxonomyintended to be useful to practitioners building similar systems.

Fields of Study & Subjects

Network Security and Intrusion Detection Adversarial Robustness in Machine Learning Machine Learning and Data Classification

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
23 March 2026
Language
EN