About Michelle Lin, MD

ALiEM Founder and CEO
Professor and Digital Innovation Lab Director
Department of Emergency Medicine
University of California, San Francisco

Reading from the Silver Linings Playbook: The ALiEM Connect Project

ALiEM Connect graduation

It feels like yesterday that we were sheltered-in-place, staring at our computers, wondering, “So now what?” 

As COVID-19 paused all in-person educational sessions, the early morning residency conference we used to begrudgingly join quickly became something that we profoundly missed. While we can now be “present” while wearing sweatpants and a button-down shirt, we miss the human connection. Many of us would gladly even suffer through traffic just to be a part of this morning conference tradition.

As educators and innovators, we know what a disruptive force the COVID-19 pandemic has been to the medical community. It has strained our medical and healthcare systems and has irrevocably altered our day-to-day lives. Without a doubt, the pandemic also changed how we delivered educational content to our learners over the past year.

Scholars have written about how likely this pandemic will likely precipitate the much-needed digital transformation of healthcare and health professions education that many of us have expected and hoped for. But while some of these innovations are born out of necessity, they may also inadvertently isolate us from the experiential aspects of education and human interaction that provide meaning to our work. For the ALiEM team, we cherish the opportunity to be part of some of these significant innovative and positive “disruptions,” further aligning our goal of creating an impactful and fulfilling academic life in emergency medicine. 

The Backstory

As a remote team working across continents, the ALiEM team has thrived on digital connection for over a decade. With excellent collaborators and volunteers representing different parts of the world, our daily operations require us to stay connected and work asynchronously to achieve our goals and deliverables. When the lockdowns hit, we leveraged its impact on physical distancing and leaned into connecting with each other even more! They say “chance favors the prepared mind,” and there we were, already on Slack and yearning for the opportunity to harness the power of teamwork using our shared passions, individual creative strengths, and enthusiastic and supportive emojis. There were moments of creating, moments of celebration, and moments of simply being with each other – often through an evening #WifiAndWine.

By the Ides of March 2020, an auspicious time indeed, we knew we were at a turning point. Our friends and work families had been working on the front lines combating the pandemic locally, gathering PPE, and studying the effects of a virus we knew next to nothing about. New information was coming in daily, and the signal-to-noise ratio was low. In some ways, to escape the disruptions going on all around us, we banded together to focus our unique energies toward creating something as novel as the virus itself in the realm of free open-access medical education.

At a time where everyone was feeling alone, we asked ourselves how we could support the joy of learning from and with each other? In truly whirlwind fashion, the first ALiEM Connect conference went from idea to execution in less than 2 weeks, a record-breaking time even for ALiEM. Thank especially to the American Board of Emergency Medicine for sponsoring these events.

We recently made it to the semi-finals at the CORD/ACEP Innovator of the Year competition, where we shared the below video capturing the fun, collaboration, and innovative outcome of our efforts. Oh, and the familiar ratatat of Slack.

Making this a Multiple Win

The secret sauce of the ALiEM team is that we have a diverse group of people, each of whom brings their own perspective and that we are able to share with one another liberally. Dr. Michelle Lin encouraged an environment that is psychologically safe and supportive since the inception of the ALiEM enterprise. It is out of this space that our diverse team was able to successfully bring a massively successful project to fruition amid a global pandemic. What started as a small brainstorming session blossomed into ALiEM Connect – 3 distinct remote conferences featuring nationally-recognized educators and thought leaders enjoyed by residents across the country.

It’s difficult to express as a linear narrative, but looking back, it seems as though our team divided into unique roles without a second thought. Just like a production company, we had the front and back of the house. Those in the front made sure to help get people in the seats to watch; stage managers and coordinators ensured that every part of each of the ALiEM Connect experiences was phenomenally smooth. We had talented individuals who acted as hosts and speakers to ensure that each of these experiences was top-notch and engaging. In the back, Drs. Mary Haas, Yusuf Yilmaz, and Teresa Chan sprung quickly into action to create a program evaluation strategy for our ALiEM Connect program, including a formal institutional review board exemption! All the while, testing and vetting platforms and methods to distribute the material were ongoing. We built upon each technological skill, learned new platforms, and trialed different features. We had barely decided on an open, free, and accessible platform (which was, in fact, no individual platform but an amalgamation of many!) before sending out the invites.

But the fun didn’t stop there! We’re the “academic” life in emergency medicine! How could we not also share our results with the traditional academic community? Within days of finishing our first ALiEM Connect experience, our program evaluation team generated the scaffolding of a manuscript to put together our thoughts and analyze the evaluation data collected. We harnessed the power of metrics from social media platforms (YouTube, Slack, Twitter), website analytics, and end-user experiences. Harnessing all of these analytics and communicating the right message with our academic medicine community was important to inform and help others to replicate similar approaches to their residents. Our team used ready to use metrics which came from YouTube analytics. But we did not stop there as we needed more reports of how the residents and programs interacted during the Connect events in the backchannel, Slack. We developed Python supported software to export and analyze all the messages happening in separate channels. We developed a “Emoji Cloud” to see how the reactions happened, and closely analyzed the messages during the event.

Given the true novelty of the experience, we figured we might as well shoot for the moon, as they say, by submitting our innovation description paper to Academic Medicine. After all, even if they didn’t accept it, we might get some constructive reviews, to say the least. As innovators, we are comfortable with the possibility of failure. We understand the value of the saying, “You miss 100% of the shots you don’t take,” and were prepared to accept “no” as an answer. With that, we took a calculated risk, making use of the same collaborative strategy to craft a manuscript, and clicked submit.

…And we’re glad we took that shot! We are excited to share that what we sent was indeed accepted and express our gratitude for the chance to share our low-cost approach to a large-scale, nationwide residency conference! You may read the Published Ahead-of-Print version of our paper.

Moral of the story…

You might be asking yourself, “What’s the moral of the story here? Of course, with enough academics and experts, yeah, you got a paper published. Cool…” But the papers aren’t the point. In fact, during the COVID-19 pandemic, more papers have been published than ever before – more research is being done, and our whole field is changing. The point is… this is how we got to ENJOY the academic life during a pandemic! We made lemonade (and several other desserts!) out of the lemons we were handed. New knowledge comes from thinking big and trying new things. Turns out, sometimes you also have to write about those experiences and share them with others.

As emergency physicians, we know we’re good in a crisis. But this experience reminded us that by surrounding ourselves with amazing people, we could get a surprising amount of work done (at record speed) and have a fantastically memorable time along the way. The moral of this story is that when you bring great people together and give them a chance to get to know each other, magic happens. ALiEM Connect happens. And we impact more people than we can possibly meet at the touch of our keyboards. We are so grateful for the chance to work alongside all the wonderful people at each of our institutions every day. Still, also, we are indebted to those who are our digital family. Thank you to all of you who make initiatives like ALiEM Connect possible. Academic life in emergency medicine is all about bringing a great team together.

So is the ALiEM team.

Bridge to EM Curriculum: Test your learning at ALiEMU

bridge to EM badges EMRA

Do you know of a senior Emergency Medicine (EM) medical student about to start internship? We are thrilled to announce that our 8-week, self-paced Bridge to EM learning curriculum has been upgraded with free assessment quizzes, badges, and certificates on ALiEMU. This curriculum, created by Drs. Timothy Wetzel and Christina Shenvi as we all shifted learning to online modalities, has garnered over 100,000 views in less than 10 months and was featured by the Association of American Medical Colleges. This bundled learning experience was created in partnership with our sponsor, Emergency Medicine Residents’ Association (EMRA).

Bridge to EM ALiEMU course grid

Go to the ALiEMU Bridge to EM Series.

Test your knowledge and see if you can get the mega-badge of EM Bridge Crosser, when you successfully pass all 9 Bridge quizzes and attain 64 hours of ALiEMU learning credit.

Sign up for the ALiEMU newsletter for announcements of any new ALiEMU courses.

Sign up for the ALiEMU Coaches newsletter to keep apprised of any changes to the coaching and Educator Dashboard features.

By |2021-03-23T10:41:39-07:00Apr 1, 2021|Academic, ALiEMU|

Computerized Adaptive Screen for Suicidal Youth (CASSY) study

CASSY PECARN suicide screening tool

Adolescent suicide rates in the United States, partly augmented by the COVID-19 pandemic, are steadily increasing [1, 2]. A commonly used screening tool is the 4-question Ask Suicide-Screening Questions (ASQ) instrument, which has a sensitivity and specificity of 60% and 92.7%, respectively, in predicting suicide-related events within 3 months. This was derived from a retrospective study of 15,003 pediatric patients (age 10-18 years) [3]. Given the morbidity and mortality associated with suicide attempts, is there a better screening tool with a higher sensitivity than 60%, while also maintaining adequate specificity? A higher sensitivity rate ensures that we have fewer misses.

The CASSY tool

In JAMA Psychiatry 2021, the Pediatric Emergency Care Applied Research Network (PECARN) researchers report derivation and external validation data for their suicide screening tool, called the Computerized Adaptive Screen for Suicidal Youth (CASSY) [4]. This publication was actually two studies in one: a derivation of the tool and then an external validation.

Terminology

This paper assumes that the reader understands certain predictive analytics methodologies and test design concepts. Let’s briefly review some of the foundational terminology used:

  • Item response theory [Wikipedia]: “It is a theory of testing based on the relationship between individuals’ performances on a test item and the test takers’ levels of performance on an overall measure of the ability that item was designed to measure.” Of note, each item may be weighted differently based on how well it correlates with the overall outcome measure, which in this study was suicide attempt within 3 months.
  • Computerized adaptive testing [Wikipedia]: This computer testing strategy, also known as tailored testing, presents questions based on the individual’s response to a prior question.
  • Receiver operator characteristics (ROC): “The performance of a diagnostic test in the case of a binary predictor can be evaluated using the measures of sensitivity and specificity. However, in many instances, we encounter predictors that are measured on a continuous or ordinal scale. In such cases, it is desirable to assess performance of a diagnostic test over the range of possible cutpoints for the predictor variable. This is achieved by a receiver operating characteristic (ROC) curve that includes all the possible decision thresholds from a diagnostic test result.” [5] In other words, test sensitivities can be calculated for set specificities of, for instance, 70%, 80%, and 90%. Based on the purpose of the diagnostic test, the binary predictor threshold would be set accordingly.
  • Area under the curve (AUC): Calculating the AUC for the ROC is an effective means to determine a diagnostic test’s accuracy. The AUC ranges from 0 to 1 with 0.5 meaning no discrimination (i.e., the test can not diagnose patients with and without the disease based on the test). Generally, an AUC value of 0.7-0.8 is acceptable, 0.8 to 0.9 is excellent, and >0.9 is outstanding [5].

Study 1: CASSY derivation

A total of 6,536 adolescents (age 12-17 years) from 13 PECARN emergency departments were enrolled and a subset were randomly received follow-up in 3 months to assess for a suicide attempt. These patients responded to 92 questions on a computer tablet. Using a multidimensional item response theory approach, the more correlated questions (72) were used to create the CASSY tool.

Test characteristic results:

  • AUC: 0.89 (excellent)
  • Using the ROC curve, the CASSY sensitivity was 83% and 61% for the fixed specificity of 80% and 90%, respectively.

Study 2: CASSY validation

A total of 4,050 adolescents from 14 PECARN emergency departments were enrolled, and all received 3-month follow-up assessing for a suicide attempt. These patients completed the CASSY tool, as well as a subset of questions from study 1 for comparison. The frequency of questions used in the adaptive screen are itemized in the paper.

Test characteristic results:

  • AUC 0.87 (excellent)
  • Using the ROC curve and at the 80% specificity cutoff from study 1, the CASSY sensitivity was 82.4% and specificity was 72.5%.

CASSY figure ROC

Limitations

Although there was strong study rigor by deriving and independently validating the tool in separate, multicenter populations, it should be noted that generalizability may be affected.

  1. The study was conducted in academic pediatric emergency departments.
  2. There was quite a few patients who were lost to follow up (27.1% in study 1, 30.5% in study 2), which may have skewed the results.
  3. Selection bias may have occurred because of patients declining to participate in the study (62% enrollment rate in study 1, 62.2% in study 2)

Bottom line

The CASSY tool accurately serves as a screening predictive tool for adolescents at risk for a suicide attempt in 3 months. Rather than having patients complete exhaustively long (and practically unfeasible) screening questions in the emergency department, this computerized adaptive tool required only a mean of 11 questions, which took a median time of 1.4 minutes (IQR 0.98-2.06 minutes) to complete.

How can you implement CASSY in your emergency department?

We asked the authors this question, and the answer is in the podcast below.

Podcast

Listen more with author Dr. Jacqueline Grupp-Phelan talking with ALiEM podcast host, Dr. Dina Wallin, about this landmark paper and behind-the-scenes issues not included on the paper.

This blog post was expert peer-reviewed by Drs. King and Grupp-Phelan, who authored the paper.

References

  1. Hill RM, Rufino K, Kurian S, Saxena J, Saxena K, Williams L. Suicide Ideation and Attempts in a Pediatric Emergency Department Before and During COVID-19 [published online ahead of print, 2020 Dec 16]. Pediatrics. 2020;e2020029280. PMID: 33328339
  2. Centers for Disease Control and Prevention. Web-based Injury Statistics Query and Reporting System (WISQARS). Published 2020.
  3. DeVylder JE,Ryan TC, Cwik M, et al. Assessment of selective and universal screening for suicide risk in a pediatric emergency department. JAMA Netw Open. 2019;2(10):e1914070-e1914070. PMID 31651971
  4. King CA, Brent D, Grupp-Phelan J, et al. Prospective Development and Validation of the Computerized Adaptive Screen for Suicidal Youth [published online ahead of print, 2021 Feb 3]. JAMA Psychiatry. 2021; 10.1001/jamapsychiatry.2020.4576. doi:10.1001/jamapsychiatry.2020.4576. PMID 33533908
  5. Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol. 2010;5(9):1315-1316. doi:10.1097/JTO. 0b013e3181ec173d

Listen to all the PECARN podcasts

EM Match Advice: 2020-21 Interview Season | This is how it started, this is how it’s going

EM Match Advice residency interview season 2020-21

The 2020-21 residency interview season has required rapid innovation and adaptability for both medical student applicants and residency programs, given COVID-19’s physical distancing restrictions. Listen to how it started and how it is going thus far with podcast co-hosts Dr. Michael Gisondi (Stanford) and Dr. Michelle Lin (UCSF).

Podcast episode with 2020-21 mid-season update and insights


Program director panelists

  • Dr. Camiron Pfennig – Prisma Health University of South Carolina Greenville
  • Dr. Ryan Bodkin – University of Rochester
  • Dr. Michael Kiemeney – Loma Linda University

Listen to all the episodes of the EM Match Advice Series

Additional resources

By |2021-07-01T10:16:37-07:00Jan 15, 2021|EM Match Advice, Podcasts|

Free eBook Announcement: Emergency Medicine Resident Simulation Curriculum for Pediatrics (EM ReSCu Peds)

emergency medicine resident simulation curriculum for pediatrics EM ReSCu Peds

 

The Emergency Medicine Resident Simulation Curriculum for Pediatrics (EM ReSCu Peds) is here! This free ebook contains 16 EM resident-tested, peer reviewed cases covering essential pediatric content identified through a robust modified Delphi process [1] with experts across the United States. Each chapter contains robust supporting materials to help educators prepare, execute, and debrief cases with residents at every level to help supplement the clinical experience.

Download the EM ReSCu Peds eBook

We request some basic demographic about you and how you plan to use the educational cases in the download form to provide us with necessary insights whether there is a need for such a resource.

A National Collaborative Effort

Cases were created and iteratively peer reviewed by members of 10 organizations represented in a national collaborative of EM, PEM, and simulation experts. Participating organizations included:

  • American Academy of Emergency Medicine
  • American Academy of Pediatrics
  • American College of Emergency Physicians
  • Council of Emergency Medicine Residency Directors
  • Emergency Medicine Residents’ Association
  • International Network for Simulation-based Pediatric Innovation, Research, & Education
  • International Pediatric Simulation Society
  • Pediatric Trauma Society
  • Society for Academic Emergency Medicine
  • Society for Simulation in Healthcare

In total, EM and PEM physicians from 44 institutions participated in the development process of this educational resource aimed at preparing EM residents to care for critically ill children.

 

Reference

  1. Mitzman J, Bank I, Burns RA, et al. A Modified Delphi Study to Prioritize Content for a Simulation-based Pediatric Curriculum for Emergency Medicine Residency Training Programs. AEM Educ Train. 2019;4(4):369-378. Published 2019 Dec 12. doi:10.1002/aet2.10412
By |2021-01-09T11:52:36-08:00Jan 12, 2021|Pediatrics, Simulation|

ALiEM 2020 Annual Report: A Year of Highs, Lows, and Adaptability

The year 2020 has been overshadowed by significant adversity for our emergency medicine community, as we battle the COVID-19 pandemic and systemic racism at work and at home. I’m continually impressed by how the team has rallied to support each other, launch new initiatives to fill new gaps in the health professions education arena, and adapt to uncertain times. Our annual report summarizes some of our highlights that we are proud to share with our readership.


View our prior Annual Reports.

By |2020-12-15T09:56:44-08:00Jan 1, 2021|Annual Report|

2021-22 ALiEM Faculty Incubator applications open: Bye COVID, hello scholarship

2021 Faculty Incubator applications ALiEMThe ALiEM Faculty Incubator is opening its doors to the fifth class of 30 educator-scholars. We have been thrilled by the quality of collaboration, growth, and scholarship output from this community ever since our first class in 2016. This year, the organization is led by Dr. Sree Natesan as the Chief Operations Officer. She has an all-star leadership team which includes Dr. Sara Krzyzaniak (Chief Academic Officer), Dr. Antonia Quinn (Associate Chief Operations Officer), Dr. Michael Gottlieb (Chief Strategic Officer), and Dr. Teresa Chan (Senior Advisor). Read more about the application process on the Faculty Incubator home page.

The deadline is January 31, 2021.

By |2020-12-21T10:10:18-08:00Dec 20, 2020|Faculty Incubator, Incubators|
Go to Top