Replication Data for: Bye-bye Bluebook?

This Article is the first to empirically assess AI’s ability to follow The Bluebook. One of the central promises of AI was that it would automate drudgery—the formulaic tasks of modern life that are time-intensive but non-discretionary. In law, few tasks better fit this description than citation. Yet, rather than prioritizing these formal, machine-checkable areas of legal practice where reliability should be easiest (and least controversial) to secure, much of the field has instead concentrated on high-stakes, open-textured domains that invite hallucination and confabulation. This approach to legal AI adoption, we argue, is upside-down. To test whether today’s AI systems can deliver on its original promise, we conduct three experiments. First, we benchmark off-the-shelf AI models on basic Bluebook tasks and find strikingly low compliance, with accuracy rates below 50 percent. Second, we evaluate whether providing the model with the relevant Bluebook rules mitigates these errors; even with this scaffolding, performance shows no meaningful improvement. Finally, in a pre-registered, practice-proximate experiment conducted in partnership with several top law reviews, we give the models the same “write-on” examinations used to select student editors, and we score the models’ performance against actual write-on candidates. Across journals, AI scores fall far below the required standard of a successful applicant. Together, these findings call into question prevailing AI hype and motivate a reorientation in AI-related legal scholarship. Before AI systems can be entrusted with judgment-laden tasks like legal interpretation, judicial sentencing, or case law research, they must first prove capable of reliably executing the formal, rule-bound processes that structure legal practice in the first place. By reorienting legal AI development along these lines, this Article proposes a pathway for legal AI adoption that advances innovation without sacrificing law’s commitment to professional integrity.