Reproducibility and credibility in computational social science: Testing the fidelity of GenAI-generated data

This study examines the fidelity of datasets generated by generative AI (GenAI) tools in comparison to real-world survey data, addressing methodological implications for computational social science. Using an original survey as a benchmark, we created three datasets: authentic respondent data, GenAI-simulated datasets produced through programmed distributional prompts, and GenAI-reconstructed datasets designed to replicate plausible human responses. Both artificial datasets were generated using large language models, specifically ChatGPT and Perplexity. Comparative analyses focused on distributional accuracy, reliability, and hypothesis-testing validity. Findings show that GenAI-simulated datasets approximated statistical distributions but lacked natural response heterogeneity, whereas GenAI-reconstructed datasets captured contextual patterns and nuanced correlations but diverged from exact statistical parameters. These results highlight the distinct yet complementary roles of AI-driven simulation and reconstruction for replication and robustness testing. The study contributes to discussions on reproducibility, credibility, and ethical integration of GenAI in high-impact social science research.