Measuring the equity of risk assessment instruments used in child protection

Abstract

Background: It is not enough for risk assessment tools to be accurate at an aggregate level; they must also operate equitably across racial groups. Objective: This paper identifies and evaluates approaches to measuring the equity of risk assessment instruments. Participants and setting: NA. Methods: The paper draws on literature in child protection as well as other fields where equity is important including employment and criminal justice. Results: Two dimensions of equity were identified. Avoidance of prejudice can be measured qualitatively by assessing the underpinning logic of a tool. Literature has focused on avoiding unconscious bias, which fails to capture both direct and indirect discrimination. Comparative validity can be measured quantitatively by comparing actual and predicted rates of recurrence for children of different groups. This can be achieved by comparing both sensitivity and specificity, or receiver operator characteristics. However, child protection literature has relied primarily on crossover, which cannot be used to make valid comparisons between groups with different overall rates of recurrence. For example, if the recurrence rate in Community A is half that in Community B (30% vs 60%), a risk assessment could avoid crossover with wide discrepancies in numbers of false positives (25% vs 1.3%) and false negatives (39.3% vs 89.4%) observed in each community. Conclusions: Current methods for measuring equity of child protection risk assessments, including crossover, are likely to exacerbate rather than ameliorate inequity. Evaluations of equity should use valid measures of comparative validity and ensure tools consider what people do, rather than who they resemble.