The Effect of Sample Heterogeneity and Risk Categorization on Area Under the Curve Predictive Validity Metrics

Forensic researchers often assume that the widely used area under the curve (AUC) predictive validity statistic can be readily compared across risk assessment instruments and between studies. From risk distributions for 224,771 convicted English and Welsh offenders, I quantify the extent to which th...

Full description

Saved in:  
Bibliographic Details
Main Author: Howard, Philip D. (Author)
Format: Electronic Article
Language:English
Published: [2017]
In: Criminal justice and behavior
Year: 2017, Volume: 44, Issue: 1, Pages: 103-120
Online Access: Volltext (Resolving-System)
Journals Online & Print:
Drawer...
Check availability: HBZ Gateway
Keywords:
Predictive validity
Violence risk assessment
Actuarial risk prediction
AUC
Area under the curve
Female offenders
OGRS
OVP
Description
Summary:Forensic researchers often assume that the widely used area under the curve (AUC) predictive validity statistic can be readily compared across risk assessment instruments and between studies. From risk distributions for 224,771 convicted English and Welsh offenders, I quantify the extent to which the AUCs of two continuously scored actuarial instruments are dependent upon sample heterogeneity and risk categorization. Sample composition can cause predictive validity to vary by 10% to 20% over chance between subpopulations, with higher AUCs for female subgroups whose scores have high variability, and lower AUCs when sampling is restricted by index offense and/or sentence type. Risk categorization has a potentially large effect on AUCs, which fall when few categories are used, especially when those categories contain unequal numbers of offenders. An improved understanding of how these factors can affect AUCs will inform professionals using risk assessment instruments and researchers comparing the validity of multiple instruments.
ISSN:1552-3594
DOI:10.1177/0093854816678899