Adolescent suicide rates in the United States, partly augmented by the COVID-19 pandemic, are steadily increasing [1, 2]. A commonly used screening tool is the 4-question Ask Suicide-Screening Questions (ASQ) instrument, which has a sensitivity and specificity of 60% and 92.7%, respectively, in predicting suicide-related events within 3 months. This was derived from a retrospective study of 15,003 pediatric patients (age 10-18 years) [3]. Given the morbidity and mortality associated with suicide attempts, is there a better screening tool with a higher sensitivity than 60%, while also maintaining adequate specificity? A higher sensitivity rate ensures that we have fewer misses.

The CASSY tool

In JAMA Psychiatry 2021, the Pediatric Emergency Care Applied Research Network (PECARN) researchers report derivation and external validation data for their suicide screening tool, called the Computerized Adaptive Screen for Suicidal Youth (CASSY) [4]. This publication was actually two studies in one: a derivation of the tool and then an external validation.

Terminology

This paper assumes that the reader understands certain predictive analytics methodologies and test design concepts. Let’s briefly review some of the foundational terminology used:

• Item response theory [Wikipedia]: “It is a theory of testing based on the relationship between individuals’ performances on a test item and the test takers’ levels of performance on an overall measure of the ability that item was designed to measure.” Of note, each item may be weighted differently based on how well it correlates with the overall outcome measure, which in this study was suicide attempt within 3 months.
• Computerized adaptive testing [Wikipedia]: This computer testing strategy, also known as tailored testing, presents questions based on the individual’s response to a prior question.
• Receiver operator characteristics (ROC): “The performance of a diagnostic test in the case of a binary predictor can be evaluated using the measures of sensitivity and specificity. However, in many instances, we encounter predictors that are measured on a continuous or ordinal scale. In such cases, it is desirable to assess performance of a diagnostic test over the range of possible cutpoints for the predictor variable. This is achieved by a receiver operating characteristic (ROC) curve that includes all the possible decision thresholds from a diagnostic test result.” [5] In other words, test sensitivities can be calculated for set specificities of, for instance, 70%, 80%, and 90%. Based on the purpose of the diagnostic test, the binary predictor threshold would be set accordingly.
• Area under the curve (AUC): Calculating the AUC for the ROC is an effective means to determine a diagnostic test’s accuracy. The AUC ranges from 0 to 1 with 0.5 meaning no discrimination (i.e., the test can not diagnose patients with and without the disease based on the test). Generally, an AUC value of 0.7-0.8 is acceptable, 0.8 to 0.9 is excellent, and >0.9 is outstanding [5].

Study 1: CASSY derivation

A total of 6,536 adolescents (age 12-17 years) from 13 PECARN emergency departments were enrolled and a subset were randomly received follow-up in 3 months to assess for a suicide attempt. These patients responded to 92 questions on a computer tablet. Using a multidimensional item response theory approach, the more correlated questions (72) were used to create the CASSY tool.

Test characteristic results:

• AUC: 0.89 (excellent)
• Using the ROC curve, the CASSY sensitivity was 83% and 61% for the fixed specificity of 80% and 90%, respectively.

Study 2: CASSY validation

A total of 4,050 adolescents from 14 PECARN emergency departments were enrolled, and all received 3-month follow-up assessing for a suicide attempt. These patients completed the CASSY tool, as well as a subset of questions from study 1 for comparison. The frequency of questions used in the adaptive screen are itemized in the paper.

Test characteristic results:

• AUC 0.87 (excellent)
• Using the ROC curve and at the 80% specificity cutoff from study 1, the CASSY sensitivity was 82.4% and specificity was 72.5%.

Limitations

Although there was strong study rigor by deriving and independently validating the tool in separate, multicenter populations, it should be noted that generalizability may be affected.

1. The study was conducted in academic pediatric emergency departments.
2. There was quite a few patients who were lost to follow up (27.1% in study 1, 30.5% in study 2), which may have skewed the results.
3. Selection bias may have occurred because of patients declining to participate in the study (62% enrollment rate in study 1, 62.2% in study 2)

Bottom line

The CASSY tool accurately serves as a screening predictive tool for adolescents at risk for a suicide attempt in 3 months. Rather than having patients complete exhaustively long (and practically unfeasible) screening questions in the emergency department, this computerized adaptive tool required only a mean of 11 questions, which took a median time of 1.4 minutes (IQR 0.98-2.06 minutes) to complete.

How can you implement CASSY in your emergency department?

We asked the authors this question, and the answer is in the podcast below.

Podcast

Listen more with author Dr. Jacqueline Grupp-Phelan talking with ALiEM podcast host, Dr. Dina Wallin, about this landmark paper and behind-the-scenes issues not included on the paper.

This blog post was expert peer-reviewed by Drs. King and Grupp-Phelan, who authored the paper.

References

1. Hill RM, Rufino K, Kurian S, Saxena J, Saxena K, Williams L. Suicide Ideation and Attempts in a Pediatric Emergency Department Before and During COVID-19 [published online ahead of print, 2020 Dec 16]. Pediatrics. 2020;e2020029280. PMID: 33328339
2. Centers for Disease Control and Prevention. Web-based Injury Statistics Query and Reporting System (WISQARS). Published 2020.
3. DeVylder JE,Ryan TC, Cwik M, et al. Assessment of selective and universal screening for suicide risk in a pediatric emergency department. JAMA Netw Open. 2019;2(10):e1914070-e1914070. PMID 31651971
4. King CA, Brent D, Grupp-Phelan J, et al. Prospective Development and Validation of the Computerized Adaptive Screen for Suicidal Youth [published online ahead of print, 2021 Feb 3]. JAMA Psychiatry. 2021; 10.1001/jamapsychiatry.2020.4576. doi:10.1001/jamapsychiatry.2020.4576. PMID 33533908
5. Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol. 2010;5(9):1315-1316. doi:10.1097/JTO. 0b013e3181ec173d

Listen to all the PECARN podcasts