Type
Article
Keywords
black box studies, error rates, firearms, fingerprints, logistic regression
Abstract
In the USA, ‘black box’ studies are increasingly being used to estimate the error rate of forensic disciplines. A sample of forensic examiner participants is asked to evaluate a set of items whose source is known to the researchers but not to the participants. Participants are asked to make a source determination (typically an identification, exclusion, or some kind of inconclusive). We study inconclusives in two black box studies, one on fingerprints and one on bullets. Rather than treating all inconclusive responses as functionally correct (as is the practice in reported error rates in the two studies we address), irrelevant to reported error rates (as some would do), or treating them all as potential errors (as others would do), we propose that the overall pattern of inconclusives in a particular black box study can shed light on the proportion of inconclusives that are due to examiner variability. Raw item and examiner variances are computed and compared with the results of a logistic regression model that takes account of which items were addressed by which examiner. The error rates reported in black box studies are substantially smaller than ‘failure rate’ analyses that take inconclusives into account. The magnitude of this difference is highly dependent on the particular study at hand.
Language
English
Department(s)
Mathematics and Statistics
Journal or Book Title
Law, Probability and Risk
Publication Year
2026
Issue Month/Season
January
Page range
1-20
DOI
https://doi.org/10.1093/lpr/mgag001
Publisher
Oxford University Press
Rights Management
The original authors own the copyright to this work and have granted Carleton College permission to display and distribute it online through the Carleton College Library. For more information on the copyright status of this work, refer to the current copyright holder.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Recommended Citation
Luby, A., & Kadane, J. B. (2026). A variance decomposition approach to inconclusives in forensic black box studies. Law, Probability and Risk, 25 (1) Available at: https://doi.org/https://doi.org/10.1093/lpr/mgag001. Accessed via Faculty Work. Mathematics. Carleton Digital Commons. https://digitalcommons.carleton.edu/math_faculty/4
The definitive version is available at https://doi.org/https://doi.org/10.1093/lpr/mgag001
