Comment on “A Variance Decomposition Approach to Inconclusives in Forensic Black Box Studies” by Amanda Luby and Joseph Kadane

It is a pleasure to read the novel approach that Luby and Kadane offer (hereafter, L + K) to address a familiar problem in reporting the results from forensic “black-box” studies: how to incorporate “inconclusive” (INC) decisions into reported error rates (FP = false positives and FN = false negatives). Forensic analyses may be presented in a courtroom; consequently, all stakeholders must have reliable information on their error rates. The forensic discipline especially needs careful studies to estimate error rates, so judges and juries can know when to, or not to, trust the results when a defendant’s freedom is at stake. As the authors note, the presence of “inconclusive” decisions can turn a simple error rate calculation into a challenging problem, because reported error rates can greatly depend on how “inconclusives” are counted. The NAS report (National Research Council 2009) called attention to the problem of inconclusives in calculating error rates for latent print analysis specifically. The report did not recommend a method for adjusting for inconclusives, in part because very few studies of error rates, in latent print analysis or in any other forensic discipline, had been published in the peer-reviewed literature at the time of the report. The report did recommend that “By documenting the relevant information gathered during the analysis, evaluation, and comparison of latent prints and the basis for the conclusion (identification, exclusion, or inconclusive), the examiner will create a transparent record of the method and thereby provide the courts with additional information on which to assess the reliability of the method for a specific case” (p. 143).