Predicting Violence and Antisocial Behavior
Predicting Violence and Antisocial Behavior
This systematic review and meta-analysis examined the predictive validity of violence risk assessment tools from 73 samples involving 24,847 individuals in 13 countries. Our principal finding was that there was heterogeneity in the performance of these measures depending on the purpose of the risk assessment. If used to inform treatment and management decisions, then these instruments performed moderately well in identifying those individuals at higher risk of violence and other forms of offending. However, if used as sole determinants of sentencing, and release or discharge decisions, these instruments are limited by their positive predictive values: 41% of people judged to be at moderate or high risk by violence risk assessment tools went on to violently offend, 23% of those judged to be at moderate or high risk by sexual risk assessment tools went on to sexually offend, and 52% of those judged to be at moderate or high risk by generic risk assessment tools went on to commit any offence. In samples with lower base rates than those that contributed to the review, such as in general psychiatry, positive predictive values will probably be even lower. However, negative predictive values were high, and suggest that these tools can effectively screen out individuals at low risk of future offending. Whether the cautious optimism that experts have described in relation to the ability to predict violence seems justified will depend on the use to which these instruments are put.
Any comparison of these risk assessment scores with other common medical diagnostic and prognostic tools poses several difficulties. Firstly, comparison with diagnostic tools is mostly inappropriate because risk assessment instruments attempt to predict the likelihood of a future outcome, whereas diagnostic instrument attempt to detect the presence of a current condition. Secondly, although it may be possible to compare performance statistics of these tools with those estimating, for example, cardiovascular risk, the implications of positive predictive values need to be considered in evaluating any comparisons. Violence risk assessment potentially leads to detention of individuals for longer than necessary, with its related economic, social, and civil rights consequences. By comparison with common medical prognostic tools, it is possible to argue that the predictive accuracy of violence risk assessment needs to be higher because of these consequences, which extend beyond the person to other people. On the other hand, it is precisely because of the risks to other people that low positive predictive values may not be as important as the ability of these instruments to predict those that are not at risk. Our introduction of a novel performance measure, the number safely discharged, could help quantify this in future research.
Despite these caveats, the areas under the curve found in this review (0.66 to 0.74) were not dissimilar to those found in studies examining scores from the most validated cardiovascular risk scheme in predicting cardiovascular disease events. Areas under the curve from the Framingham scoring system range from 0.57 to 0.86, the SCORE from 0.65 to 0.85, and QRISK from 0.76 to 0.79. Many of these studies report associations between predicted and observed risks, which may be helpful for future research in violence risk assessment. Finally, the standard by which these instruments are compared will differ depending on their setting. In forensic psychiatry, a more meaningful comparison will be with unstructured clinical judgment, and clinical trials are needed to test whether structured risk assessment reduces adverse outcomes.
One implication of these findings is that, even after 30 years of development, the view that violence, sexual, or criminal risk can be predicted in most cases is not evidence based. This message is important for the general public, media, and some administrations who may have unrealistic expectations of risk prediction for clinicians. This expectation is not as high in other medical specialties, in which the expectation that the doctor will identify the individual patient who will have an adverse event is not a primary issue whereas psychiatry, in many countries such as the UK, has developed a culture of inquiries.
A second and related implication is that these tools are not sufficient on their own for the purposes of risk assessment. In some criminal justice systems, expert testimony commonly use scores from these instruments in a simplistic way to estimate an individual’s risk of serious repeat offending. However, our review suggests that risk assessment tools in their current form can only be used to roughly classify individuals at the group level, and not to safely determine criminal prognosis in an individual case. This approach is mostly used in forensic psychiatry in the UK and other western countries, where they form part of a wider clinical assessment process. These instruments may also assist in developing risk management plans in selected high risk groups, as suggested by recent clinical guidelines in England and Wales. Furthermore, they are preferable to unstructured clinical judgment owing to their increased transparency and reliability.
Another implication is that actuarial instruments focusing on historical risk factors perform no better than tools based on clinical judgment, a finding contrary to some previous reviews. Finally, our review suggests that these instruments should be used differently. Since they had higher negative predictive values, one potential approach would be to use them to screen out low risk individuals. Researchers and policy makers could use the number safely discharged to determine the potential screening use of any particular tool, although its use could be limited for clinicians depending on the immediate and service consequences of false positives. A further caveat is that specificities were not high—therefore, although the decision maker can be confident that a person is truly low risk if screened out, when someone fails to be screened out as low risk, doctors cannot be certain that this person is not low risk. In other words, many individuals assessed as being at moderate or high risk could be, in fact, low risk. Ultimately, however, what constitutes an appropriate balance between the ethical implications of detaining people based on the predictive ability of these tools and the need for public protection will primarily be a political consideration.
Previous meta-analyses on risk assessment have focused on comparing instruments with one another, or measuring how individual tools perform across sexes and ethnic groups. A systematic review published in 2001 examined the accuracy of violence risk assessment in high risk groups, and was based on 21 studies. It estimated that six people needed to be detained to prevent one violent offence, compared with our current review’s estimate of two people needing detention. This difference was despite the median base rate of violence being similar in both reviews (current review, 32% (interquartile range 22-46%) v 2001 review, 26%, 15-41%). Unlike the previous report, the present meta-analysis focused on structured assessment instruments and included both institutional and community samples. The current report reviewed more than three times as many studies as the 2001 review and a recent meta-analysis that only compared head to head investigations of tool use.
The strengths of the current review include the incorporation of new tabular data, the reporting of multiple accuracy estimates, and a meta-analysis using bivariate models. We received new tabular data for 14,798 people (60% of people included in the review), and hence have reported a considerable amount of new data. Finally, by using a range of accuracy estimates, we have attempted to minimise biases that may be associated with reporting only one of them.
Limitations include that we solely examined the predictive qualities of these risk assessment tools, and did not account for their potential role in informing management and reminding clinicians to enquire about potentially important prognostic and modifiable factors. In addition, we found moderate to high levels of heterogeneity. Heterogeneity was to be expected, in view of the different types of samples included in the primary studies (from prison, secure hospitals, and general psychiatric hospitals) and outcomes measured. We explored sources of heterogeneity and found no clear trends. Investigating heterogeneity in diagnostic odds ratios meant that incidence of the outcome was accounted for. One possible source of heterogeneity was the potential effects of intervention after a risk assessment, particularly in people deemed high risk. We compared diagnostic odds ratios between prospective and retrospective studies that would be expected, to some extent, to measure this, since high risk participants identified in prospective studies would probably have been enrolled in interventions designed to reduce violence risk. However, we found no differences in metaregression or subgroup analysis. Nevertheless, clinical trials are needed directly to test the possible effects of intervention. Although we tested for publication status and found no clear patterns, we cannot exclude the possibility that such bias could exist in the studies that we were unable to include. Registers of such investigations would assist future reviews. In addition, few samples reported on women and, thus, this review was underpowered to examine whether predictive validity was different from men.
Discussion
This systematic review and meta-analysis examined the predictive validity of violence risk assessment tools from 73 samples involving 24,847 individuals in 13 countries. Our principal finding was that there was heterogeneity in the performance of these measures depending on the purpose of the risk assessment. If used to inform treatment and management decisions, then these instruments performed moderately well in identifying those individuals at higher risk of violence and other forms of offending. However, if used as sole determinants of sentencing, and release or discharge decisions, these instruments are limited by their positive predictive values: 41% of people judged to be at moderate or high risk by violence risk assessment tools went on to violently offend, 23% of those judged to be at moderate or high risk by sexual risk assessment tools went on to sexually offend, and 52% of those judged to be at moderate or high risk by generic risk assessment tools went on to commit any offence. In samples with lower base rates than those that contributed to the review, such as in general psychiatry, positive predictive values will probably be even lower. However, negative predictive values were high, and suggest that these tools can effectively screen out individuals at low risk of future offending. Whether the cautious optimism that experts have described in relation to the ability to predict violence seems justified will depend on the use to which these instruments are put.
Comparisons With Other Medical Tools
Any comparison of these risk assessment scores with other common medical diagnostic and prognostic tools poses several difficulties. Firstly, comparison with diagnostic tools is mostly inappropriate because risk assessment instruments attempt to predict the likelihood of a future outcome, whereas diagnostic instrument attempt to detect the presence of a current condition. Secondly, although it may be possible to compare performance statistics of these tools with those estimating, for example, cardiovascular risk, the implications of positive predictive values need to be considered in evaluating any comparisons. Violence risk assessment potentially leads to detention of individuals for longer than necessary, with its related economic, social, and civil rights consequences. By comparison with common medical prognostic tools, it is possible to argue that the predictive accuracy of violence risk assessment needs to be higher because of these consequences, which extend beyond the person to other people. On the other hand, it is precisely because of the risks to other people that low positive predictive values may not be as important as the ability of these instruments to predict those that are not at risk. Our introduction of a novel performance measure, the number safely discharged, could help quantify this in future research.
Despite these caveats, the areas under the curve found in this review (0.66 to 0.74) were not dissimilar to those found in studies examining scores from the most validated cardiovascular risk scheme in predicting cardiovascular disease events. Areas under the curve from the Framingham scoring system range from 0.57 to 0.86, the SCORE from 0.65 to 0.85, and QRISK from 0.76 to 0.79. Many of these studies report associations between predicted and observed risks, which may be helpful for future research in violence risk assessment. Finally, the standard by which these instruments are compared will differ depending on their setting. In forensic psychiatry, a more meaningful comparison will be with unstructured clinical judgment, and clinical trials are needed to test whether structured risk assessment reduces adverse outcomes.
Clinical Implications
One implication of these findings is that, even after 30 years of development, the view that violence, sexual, or criminal risk can be predicted in most cases is not evidence based. This message is important for the general public, media, and some administrations who may have unrealistic expectations of risk prediction for clinicians. This expectation is not as high in other medical specialties, in which the expectation that the doctor will identify the individual patient who will have an adverse event is not a primary issue whereas psychiatry, in many countries such as the UK, has developed a culture of inquiries.
A second and related implication is that these tools are not sufficient on their own for the purposes of risk assessment. In some criminal justice systems, expert testimony commonly use scores from these instruments in a simplistic way to estimate an individual’s risk of serious repeat offending. However, our review suggests that risk assessment tools in their current form can only be used to roughly classify individuals at the group level, and not to safely determine criminal prognosis in an individual case. This approach is mostly used in forensic psychiatry in the UK and other western countries, where they form part of a wider clinical assessment process. These instruments may also assist in developing risk management plans in selected high risk groups, as suggested by recent clinical guidelines in England and Wales. Furthermore, they are preferable to unstructured clinical judgment owing to their increased transparency and reliability.
Another implication is that actuarial instruments focusing on historical risk factors perform no better than tools based on clinical judgment, a finding contrary to some previous reviews. Finally, our review suggests that these instruments should be used differently. Since they had higher negative predictive values, one potential approach would be to use them to screen out low risk individuals. Researchers and policy makers could use the number safely discharged to determine the potential screening use of any particular tool, although its use could be limited for clinicians depending on the immediate and service consequences of false positives. A further caveat is that specificities were not high—therefore, although the decision maker can be confident that a person is truly low risk if screened out, when someone fails to be screened out as low risk, doctors cannot be certain that this person is not low risk. In other words, many individuals assessed as being at moderate or high risk could be, in fact, low risk. Ultimately, however, what constitutes an appropriate balance between the ethical implications of detaining people based on the predictive ability of these tools and the need for public protection will primarily be a political consideration.
Comparison With Other Studies
Previous meta-analyses on risk assessment have focused on comparing instruments with one another, or measuring how individual tools perform across sexes and ethnic groups. A systematic review published in 2001 examined the accuracy of violence risk assessment in high risk groups, and was based on 21 studies. It estimated that six people needed to be detained to prevent one violent offence, compared with our current review’s estimate of two people needing detention. This difference was despite the median base rate of violence being similar in both reviews (current review, 32% (interquartile range 22-46%) v 2001 review, 26%, 15-41%). Unlike the previous report, the present meta-analysis focused on structured assessment instruments and included both institutional and community samples. The current report reviewed more than three times as many studies as the 2001 review and a recent meta-analysis that only compared head to head investigations of tool use.
Strengths and Limitations
The strengths of the current review include the incorporation of new tabular data, the reporting of multiple accuracy estimates, and a meta-analysis using bivariate models. We received new tabular data for 14,798 people (60% of people included in the review), and hence have reported a considerable amount of new data. Finally, by using a range of accuracy estimates, we have attempted to minimise biases that may be associated with reporting only one of them.
Limitations include that we solely examined the predictive qualities of these risk assessment tools, and did not account for their potential role in informing management and reminding clinicians to enquire about potentially important prognostic and modifiable factors. In addition, we found moderate to high levels of heterogeneity. Heterogeneity was to be expected, in view of the different types of samples included in the primary studies (from prison, secure hospitals, and general psychiatric hospitals) and outcomes measured. We explored sources of heterogeneity and found no clear trends. Investigating heterogeneity in diagnostic odds ratios meant that incidence of the outcome was accounted for. One possible source of heterogeneity was the potential effects of intervention after a risk assessment, particularly in people deemed high risk. We compared diagnostic odds ratios between prospective and retrospective studies that would be expected, to some extent, to measure this, since high risk participants identified in prospective studies would probably have been enrolled in interventions designed to reduce violence risk. However, we found no differences in metaregression or subgroup analysis. Nevertheless, clinical trials are needed directly to test the possible effects of intervention. Although we tested for publication status and found no clear patterns, we cannot exclude the possibility that such bias could exist in the studies that we were unable to include. Registers of such investigations would assist future reviews. In addition, few samples reported on women and, thus, this review was underpowered to examine whether predictive validity was different from men.