Cleaning up the Mess: Re-Evaluating the Real-System Modeling Accuracy of Ramulator 2.0

A MICRO 2024 best paper runner-up publication (the Mess paper [1]) with all three artifact badges awarded (including “Reproducible”) proposes a new benchmark to evaluate real and simulated memory system performance. The publication contends that Ramulator 2.0 [2] and DAMOV [3] (ZSim+Ramulator) (along with other existing memory system simulators) “poorly resemble the actual system performance” and asserts that their simulator is better.In this paper, we show that the Mess paper [1] has 1) demonstrable technical misconfigurations, 2) methodological errors in interpreting simulation statistics, and 3) an incomplete artifact that makes its key results irreproducible. We demonstrate that the Ramulator 2.0 simulation results reported in [1] are incorrect due to multiple configuration errors instead of inherent simulation inaccuracy claimed by the Mess paper. We show that by correctly configuring Ramulator 2.0, Ramulator 2.0’s simulated memory system performance actually resembles real system characteristics well, and thus a key claimed contribution of [1] is factually incorrect. We also identify that the DAMOV simulation results in [1] use wrong simulation statistics that are unrelated to the simulated DRAM performance. By using correct DRAM simulation statistics, we show that DAMOV’s simulated DRAM latency is not constant, in contrast to [1]’s claim. Moreover, the Mess paper’s artifact repository [4, 5] lacks the necessary sources (simulator code, system configurations, memory traces, etc.) to fully reproduce all the Mess paper’s results. We find that the experiment scripts in [4, 5] use simulator executables and other resources that are neither described in the Mess paper nor found in the artifact repository [4, 5].Our work identifies important issues in [1]’s memory simulator evaluation methodology regarding Ramulator 2.0 and DAMOV. We present results that validate the real-system modeling accuracy of Ramulator 2.0, and describe the reasons why [1]’s results with respect to these two simulators are incorrect. We emphasize the importance of carefully and rigorously validating simulation results to avoid publishing factually incorrect results and contributions. We strongly encourage the computer architecture community to consider our corrections to the Ramulator 2.0 and DAMOV results of the Mess paper [1] to prevent the propagation of inaccurate and misleading results and to maintain the reliability of the scientific record. Our investigation also aims to stimulate discussion on artifact evaluation practices and on mechanisms for correcting results and artifacts after publication. To aid future works and reproduction of all our results, we open source all our code and scripts. We also discuss best practices and add sanity checks to aid users and developers of simulation tools.