Optimized mutation scheduling for fuzzing based on estimation of distribution algorithm

Abstract In the discovery of modern software vulnerability, mutation-based fuzzing techniques are widely applied, with their performance highly dependent on the effectiveness of mutation scheduling strategies. Most current research primarily focuses on optimizing seed scheduling. However, mutation scheduling plays an equally critical role in fuzzing, as it determines how mutation operators are selected and applied to generate new test cases. Existing mutation scheduling schemes are faced with several issues, such as the need to input manual parameters from users and improper overhead management. To address these challenges, this paper proposes an innovative mutation scheduling model, HavocEDA, utilizing Estimation of Distribution Algorithm to optimize the selection of mutation operators in fuzzing, thereby enhancing the efficiency of vulnerability detection. The HavocEDA model can dynamically adjust the probability distribution of mutation operators based on fuzzing feedback, enabling a more efficient exploration of potential vulnerabilities within the software. We implemented a prototype based on the popular general-purpose fuzzer AFL. We evaluated HavocEDA on 9 open-source Linux programs in FuzzBench. Experimental results indicate that HavocEDA shows a performance improvement in edge coverage and crash discovery compared with the state-of-the-art fuzzers AFL, DARWIN, MOpt, and HavocMAB.