Experimental and Model-Based Evaluation and Prediction of Data Distribution Service Performance

The Data Distribution Service (DDS) specification is used in many real-time, mission-critical, and distributed systems e.g. Air Traffic Control (ATC), autonomous vehicle fleets, and medical devices. Setting up these systems and configuring the components and how they communicate is not easy because of the vast number of options that can be explored. While performance is a non-functional requirement for such systems, it plays a crucial role in the selection of configuration values. System designers want to select the configuration which produces the most ideal performance (lowest latency and highest throughput) throughout the run-time of the system. To do this, one must run performance experiments for varied configurations to capture the behaviour of the system under different scenarios to find the most ideal configuration. This task is laborious and requires immense time, computational power, and effort. We contributed to this problem with a two-pronged approach. First, we executed over 6,000 performance experiments on 3 different testbeds (both virtualised and physical), to evaluate the impact of various configuration parameters, such as Multicast (MC) and data length, on the performance of DDS in terms of latency and throughput, considered as the two most important performance metrics. Second, we use collected data to be the first to train various Machine Learning (ML) models (including Linear Regression (LR), Random Forest (RF), Extreme Gradient Boosting (XGB), and Feed-Forward Neural Network (FFNN)) and assess the effectiveness of ML for predicting the performance of DDS in terms of latency and throughput, as characterised by their distributions. Our results uncover counter-intuitive findings related to MC, insights into how larger data lengths outperform smaller data lengths in terms of latency, specifically in worst-case scenarios, suitable demonstration of the applicability of ML with accuracies up to 0.99 for interpolation and several cases with accuracies above 0.9 for extrapolation, counter-intuitive findings showing that introducing noise via a bandwidth throttle improves the predictive ability of the ML models, and more.