(Un)fair Mistakes on Social Media: How Demographic Characteristics Influence Authorship Attribution

Authorship attribution techniques are increasingly being used in online contexts such as sock puppet detection, malicious account linking, and cross-platform account linking. Yet, it is unknown whether these models perform equitably across different demographic groups. Bias in such techniques could lead to false accusations, account banning, and privacy violations disproportionately impacting users from certain demographics. In this paper, we audit authorship attribution for bias in three different ways with respect to gender, native language, and age. First, we evaluate how the proportion of users with a certain demographic characteristic impacts the overall classifier performance. Second, we evaluate if a user's demographic characteristics influence the probability that their texts are misclassified. Our results for these two evaluations indicate that authorship attribution does not demonstrate bias across demographic groups in the closed-world setting. Third, we evaluate the types of errors that occur when the true author is removed from the suspect set, thereby forcing the classifier to choose an incorrect author. This controls for the influence of a users fluctuations in writing style on the mistakes made. Unlike the first two settings, our results here indicate a tendency to attribute authorship to users who share the same demographic characteristic as the true author. Our results highlight that though an NLP model may appear fair in the closed-world setting for a performant classifier, this does not guarantee fairness when errors are inevitable.