Failure Modes and Remediations in LLM-Driven Automated Machine Learning: A Case Study on Network Intrusion Detection
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
I present an empirical study of an LLM-driven automated machinelearning (AutoML) loop applied to binary network intrusion detec-tion on the CICIDS-2017 dataset. The framework adapts Karpathy’sautoresearch concept to tabular classification: Claude Sonnet 4.6iteratively rewrites a detector script, which is executed as a sub-process and evaluated against held-out validation and test sets.Over four sessions and 27 runs, I document six distinct failuremodes: revert-loop stagnation, unsupervised feature engineeringcausing train/test distribution mismatch, resource budget viola-tions, platform encoding crashes, deprecated API incompatibilities,and cross-process pickle scope errors. Concrete remediations areprovided for each. My best model achieves weighted F1 = 0.9992 onCICIDS-2017, consistent with published benchmarks. The centralfinding is that near a performance ceiling, the primary challengeof LLM-driven AutoML is not model quality but system reliabil-ity: ensuring the agent explores meaningfully, respects executionconstraints, and produces evaluable outputs. I provide a reusable,dataset-agnostic framework and a reproducible failure taxonomyintended to be useful to practitioners building similar systems.
Fields of Study & Subjects
Network Security and Intrusion Detection
Adversarial Robustness in Machine Learning
Machine Learning and Data Classification
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 23 March 2026
- Language
- EN