NEPS Technical Report for Scientific Literacy: Scaling Results of Starting Cohort 5 for Wave 5

The National Educational Panel Study (NEPS) examines the development of competencies across the lifespan. Therefore, the NEPS develops tests to assess different domains of competence in different age cohorts. To evaluate the quality of these competence tests, several analyses based on item response theory are conducted. This paper describes the data and scaling procedures for a test of scientific literacy that was administered in Wave 5 of Starting Cohort 5. The scientific literacy test consisted of 30 items representing different contexts as well as different areas of scientific knowledge. The test was administered to 8,767 participants (64% women) in an experimental design. To study the effects of different presentation modes, the test was either presented as paper-, computer-, or web-based test. A partial credit model was used to scale the data. Item fit statistics, differential item functioning, Rasch-homogeneity, test dimensionality, and local item independence were evaluated to ensure the quality of the test. These analyses showed that the items had good item fit and negligible differential item functioning across different subgroups. Furthermore, comparable measurement models could be confirmed for the different presentation modes. A limitation of the test was its restricted variance resulting in a rather poor reliability of the test scores. The estimated scores are therefore not well suited to describe individual differences in scientific literacy between respondents, despite otherwise good psychometric properties of the test. In addition to the scaling results, this report also describes the data available in the scientific use file and presents the conquest syntax for scaling the data.