Sampled social media data risk biased and inconsistent estimates of online social phenomena

There are broad concerns about the societal impacts of social media, motivating widespread research on online social phenomena. However, such research is increasingly reliant on sampled datasets, and the extent to which research using sampled data can reliably reproduce findings obtained from complete datasets is unclear. Focusing on the intersection of social media and political communication - where reliable results are especially critical - here, we investigate how sampling strategies affect estimates of political polarization and news diet credibility on Twitter/X and Reddit, two of the most commonly studied platforms. First, we show that political content is only a small subset of overall platform activity. Second, we show that large random samples can reliably measure social network polarization, but that smaller samples exhibit systematic bias and are often inconsistent. Finally, we show that engagement-weighted sampling distorts estimates of misinformation prevalence, but in an unpredictable manner which varies across platforms. Our findings underscore the risks of drawing conclusions from incomplete social media data. We argue that policy makers must urgently address this challenge, and call for the European Union's Digital Services Act to ensure that researchers have access to comprehensive social media datasets, not limited topic-specific samples, in order to ensure that social media research is accurate and informative.