The More We Measure, The Less We See: On the Replicability and Repeatability of Mobile Starlink Measurements

Lately, growing effort has been invested to analyze the Starlink performance under vehicular user terminal mobility. As such measurement campaigns are labor- and time-consuming, existing studies mostly focus on single test drives. In this paper, we demonstrate that repeatability and replicability of mobile Starlink measurements represent serious challenges, mainly due to numerous factors that can be hardly controlled. First, we conduct an extensive measurement campaign of Starlink's latency and packet loss under vehicular mobility. The campaign focuses on repeatability and consists of 35 single test drives of the identical highway route over the span of four months. Then, we demonstrate that replicability remains challenging. For this, we replicate existing findings and find new, seemingly statistically significant correlations by purposefully cherry-picking single test drives. By considering our complete dataset, we show that in reality, these findings are likely statistical variations caused by uncontrollable factors. Based on this, we discuss the challenges and provide guidelines for meaningful future Starlink measurements under vehicular mobility.