Machine Learning Prediction of Suicide Risk Does Not Identify Patients Without Traditional Risk Factors

Maricela Cruz, PhD^a,b,* ; Susan M. Shortreed, PhD^a,b ; Julie E. Richards, PhD, MPH^a,c ; R. Yates Coley, PhD^a,b ; Bobbi Jo Yarborough, PsyD^d ; Rod L. Walker, MS^a ; Eric Johnson, MS^a ; Brian K. Ahmedani, PhD^e ; Rebecca Rossom, MD, MS^f ; Karen J. Coleman, PhD^g ; Jennifer M. Boggs, PhD, MSW^h ; Arne L. Beck, PhD^h ; Gregory E. Simon, MD, MPH^a

J Clin Psychiatry 2022;83(5):21m14178

ABSTRACT

Objective: To determine whether predictions of suicide risk from machine learning models identify unexpected patients or patients without medical record documentation of traditional risk factors.

Methods: The study sample included 27,091,382 outpatient mental health (MH) specialty or general medical visits with a MH diagnosis for patients aged 11 years or older from January 1, 2009, to September 30, 2017. We used predicted risk scores of suicide attempt and suicide death, separately, within 90 days of visits to classify visits into risk score percentile strata. For each stratum, we calculated counts and percentages of visits with traditional risk factors, including prior self-harm diagnoses and emergency department visits or hospitalizations with MH diagnoses, in the last 3, 12, and 60 months.

Results: Risk-factor percentages increased with predicted risk scores. Among MH specialty visits, 66%, 88%, and 99% of visits with suicide attempt risk scores in the top 3 strata (respectively, 90th–95th, 95th–98th, and ≥ 98th percentiles) and 60%, 77%, and 93% of visits with suicide risk scores in the top 3 strata represented patients who had at least one traditional risk factor documented in the prior 12 months. Among general medical visits, 52%, 66%, and 90% of visits with suicide attempt risk scores in the top 3 strata and 45%, 66%, and 79% of visits with suicide risk scores in the top 3 strata represented patients who had a history of traditional risk factors in the last 12 months.

Conclusions: Suicide risk alerts based on these machine learning models coincide with patients traditionally thought of as high-risk at their high-risk visits.

Continue Reading...

Did you know members enjoy unlimited free PDF downloads as part of their subscription? Subscribe today for instant access to this article and our entire library in your preferred format. Alternatively, you can purchase the PDF of this article individually.

Subscribe Now

Already a member? Login

Buy PDF for $40

Please sign in or purchase this PDF for $40.

References (29)

Ahmedani BK, Simon GE, Stewart C, et al. Health care contacts in the year before suicide death. J Gen Intern Med. 2014;29(6):870–877. PubMed CrossRef
Ahmedani BK, Stewart C, Simon GE, et al. Racial/Ethnic differences in health care visits made before suicide attempt across the United States. Med Care. 2015;53(5):430–435. PubMed CrossRef
Patient Safety Advisory Group. Detecting and Treating Suicidal Ideation in all Settings. The Joint Commission Sentinel Event Alerts; 2016:56.
National Action Alliance for Suicide Prevention Transforming Health Systems Initiative Work Group. Recommended Standard Care for People With Suicide Risk: Making Health Care Suicide Safe. Washington, DC: Education Development Center, Inc.; 2018.
Simon GE, Coleman KJ, Rossom RC, et al. Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice. J Clin Psychiatry. 2016;77(2):221–227. PubMed CrossRef
Simon GE, Johnson E, Lawrence JM, et al. Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry. 2018;175(10):951–960. PubMed CrossRef
Kessler RC, Stein MB, Petukhova MV, et al. Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Mol Psychiatry. 2017;22(4):544–551. PubMed
Kessler RC, Warner CH, Ivany C, et al; Army STARRS Collaborators. Predicting suicides after psychiatric hospitalization in US Army soldiers: the Army Study To Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry. 2015;72(1):49–57. PubMed CrossRef
Barak-Corren Y, Castro VM, Javitt S, et al. Predicting suicidal behavior from longitudinal electronic health records. Am J Psychiatry. 2017;174(2):154–162. PubMed CrossRef
Walsh CG, Ribeiro JD, Franklin JC. Predicting risk of suicide attempts over time through machine learning. Clin Psychol Sci. 2017;5(3):457–469. CrossRef
Schafer KM, Kennedy G, Gallyer A, et al. A direct comparison of theory-driven and machine learning prediction of suicide: a meta-analysis. PLoS One. 2021;16(4):e0249833. PubMed CrossRef
Reger GM, McClure ML, Ruskin D, et al. Integrating predictive modeling into mental health care: an example in suicide prevention. Psychiatr Serv. 2019;70(1):71–74. PubMed CrossRef
Richards JE, Simon GE, Boggs JM, et al. An implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicide. Implement Res Pract. 2021;2:10.1177/26334895211011769. PubMed CrossRef
Velupillai S, Hadlaczky G, Baca-Garcia E, et al. Risk assessment tools and data-driven approaches for predicting and preventing suicidal behavior. Front Psychiatry. 2019;10:36. PubMed CrossRef
Belsher BE, Smolenski DJ, Pruitt LD, et al. Prediction models for suicide attempts and deaths: a systematic review and simulation. JAMA Psychiatry. 2019;76(6):642–651. PubMed CrossRef
Whiting D, Fazel S. How accurate are suicide risk prediction models? asking the right questions for clinical practice. Evid Based Ment Health. 2019;22(3):125–128. PubMed CrossRef
Reger MA, Ammerman BA, Carter SP, et al. Patient feedback on the use of predictive analytics for suicide prevention. Psychiatr Serv. 2021;72(2):129–135. PubMed CrossRef
Ross TR, Ng D, Brown JS, et al. The HMO Research Network Virtual Data Warehouse: a public data model to support collaboration. EGEMS (Wash DC). 2014;2(1):1049. PubMed CrossRef
American Psychiatric Association Work Group on Suicidal Behaviors. Practice Guideline for the Assessment and Treatment of Patients with Suicidal Behaviors. Washington, DC: American Psychiatric Association; 2010.
Department of Veterans Affairs Assessment and Management of Suicide Risk Work Group. VA/DoD Clinical Practice Guideline for the Assessment and Management of Patients at Risk for Suicide. Washington, DC: Department of Veterans Affairs; 2019.
Kroenke K, Spitzer RL, Williams JBW, et al. The Patient Health Questionnaire Somatic, Anxiety, and Depressive Symptom Scales: a systematic review. Gen Hosp Psychiatry. 2010;32(4):345–359. PubMed CrossRef
Walker RL, Shortreed SM, Ziebell RA, et al. Evaluation of electronic health record-based suicide risk prediction models on contemporary data. Appl Clin Inform. 2021;12(4):778–787. PubMed CrossRef
Charlson M, Szatrowski TP, Peterson J, et al. Validation of a combined comorbidity index. J Clin Epidemiol. 1994;47(11):1245–1251. PubMed CrossRef
Tibshirani R. Regression shrinkage and selection via the lasso. J R Stat Soc B. 1996;58(1):267–288. CrossRef
Simon GE, Shortreed SM, Johnson E, et al. What health records data are required for accurate prediction of suicidal behavior? J Am Med Inform Assoc. 2019;26(12):1458–1465. PubMed CrossRef
Belsher BE, Smolenski DJ, Pruitt LD. Positive predictive values and potential success of suicide prediction models-reply. JAMA Psychiatry. 2019;76(8):870–871. PubMed CrossRef
Simon GE, Shortreed SM, Coley RY. Positive predictive values and potential success of suicide prediction models. JAMA Psychiatry. 2019;76(8):868–869. PubMed CrossRef
Franklin JC, Ribeiro JD, Fox KR, et al. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull. 2017;143(2):187–232. PubMed CrossRef
Beam AL, Kohane IS. Big data and machine learning in health care. JAMA. 2018;319(13):1317–1318. PubMed CrossRef