Inferring Gender from Author Names with Local LLMs: A Multi-Model Evaluation

Gender identification of researchers is a common practice in scientometric studies examining inequalities in science. The most widely used approach relies on inferring gender from author names using commercial APIs or name-gender dictionaries, which often lack transparency. This study explores the use of local open-weight Large Language Models (LLMs) as an alternative for name-based gender classification. We evaluate 25 models from seven leading families (Llama, Gemma, Phi, Mistral, Qwen, DeepSeek, and Yi), ranging from 270 million to 70 billion parameters, using a reference dataset of nearly 200,000 names across 195 countries extracted from Wikidata, which allows for a systematic discussion of the size–performance relationship and of cross-family differences than is possible in other studies of the kind. Results show that top-performing models achieve F1-Scores above 0.90 for both gender categories, positioning local LLMs as a viable alternative to proprietary tools. A direct head-to-head comparison with two widely used traditional baselines (WGND 2.0 and Gender-Guesser) on the same reference dataset confirms that the best local LLMs outperform dictionary-based approaches in macro-F1, with the advantage driven primarily by coverage (≈ 95% versus ≈ 45%) rather than by per-prediction accuracy. All models exhibit systematic gender bias, showing higher precision for men and higher recall for women. Two models stand out for practical deployment: Llama3.1-8b, which combines near-top accuracy (macro-F1 ≈ 0.93) with the lowest abstention rate of the cohort (≈ 2%) and the smallest footprint among the top performers, and Mistral-Nemo-12b, which attains marginally higher macro-F1 (≈ 0.94) at a verified low operational cost. The two are practically equivalent in accuracy and differ mainly in computational footprint and coverage.