Directed or Undirected: Investigating Fuzzing Strategies in a CI/CD Setup

Fuzzing best practices suggest that fuzzing should be run for at least 24 hours, if not longer. This recommendation makes it hard to integrate fuzzing into CI/CD contexts to rapidly check a commit for bugs. Existing studies on CI/CD fuzzing found that some bugs could be reached and triggered in timeouts as short as 10 minutes, suggesting the idea is feasible. Directed fuzzers, such as AFLGo, aim to generate inputs that reach specific target locations in the program being fuzzed. Thus, they should be more effective at fuzzing in a CI/CD environment. We evaluate both directed and undirected fuzzers in a simulated CI/CD environment. We use Magma as a source of benchmarks and run fuzzers for 10 minutes. We start the fuzzing process from a corpus generated by a 24-hour fuzzing run on a previous version of the code, and run the fuzzers on versions of Magma programs with a single bug injected. As the Magma patches provide very precise bug-location information as targets to the directed fuzzers, this may give an unfair advantage to the directed fuzzers. Thus, we also conduct experiments to evaluate whether directed fuzzers are sensitive to “bloat” in commits, i.e., whether directed fuzzers can still reach the bugs if given additional target locations. Surprisingly, we find that AFL and TuneFuzz, neither of which is directed to the targets from the commits, are the fastest at reaching and triggering bugs. AFL wins if we consider the time to build and instrument the program under test. This result is explained by contrasting bug reaching/triggering time and the time to load the input corpus. We find that nearly all of the bugs reached and triggered in our experiments are reached/triggered while loading the corpus generated by the 24-hour fuzzing run, rather than while generating new inputs. This suggests that for commits as small as Magma patches, using inputs generated from a long-running fuzzing run as regression tests in CI/CD gives nearly as much bug-revealing power as conducting a short-running fuzzing session.