Replicating and expanding a validity argument for eRevise as a formative assessment: Empirical support for theories of conceptual change

As automated writing evaluation (AWE) systems proliferate, it is important to assess them for the extent to which they serve an authentic formative assessment purpose. We developed eRevise as an AWE system to engage upper elementary students in text-based writing, receive automated feedback, and revise their essay based on that feedback. In past work, we presented a validity argument for a response-to-text formative assessment. Here, we replicate evidence for the mediational processes we identified with a second response-to-text formative assessment. Beyond replication, we expand the validity argument in several ways: We examine multiple writing outcomes to understand whether the relationships in the data generalize. We also explore patterns of relationships to better understand which students are most likely to benefit from eRevise . Furthermore, we expand the investigation from feature score improvement alone to also consider students’ conceptual development over time. Our findings provide further evidence for sociocultural mechanisms supporting students’ conceptual development of evidence use in writing. We discuss the implications of our findings for future design of eRevise and AWE systems, in general. We also discuss the need for a recursive process where our findings also contribute to refined theory development and design of future AWE systems. (1). Formative Assessment; (2) Argumentative Writing; (3) Adaptive Expertise; (4) Conceptual Change; (5) Validity Argument.