New AIR Series: ALiEM Approved Instructional Resources

New AIR Series: ALiEM Approved Instructional Resources

2017-08-02T11:58:02+00:00

On behalf of the AIR Executive Board, we are excited to introduce the Approved Instructional Resources (AIR) series! The AIR series was conceived to provide a credible method by which an U.S. Emergency Medicine resident can receive academic credit for using Free Open Access Meducation (FOAM) resources. The Executive Board will release a list of high-quality FOAM educational posts and podcasts specially selected by our Executive Board, in parallel with the CORD residency training curriculum. We will have an accompanying quiz for each list and track who completes it. EM residents who complete the quiz can hopefully receive credit for Individualized Interactive Instruction (III) from their EM residency for training purposes.

Why do this?

We created this series because we have observed residencies struggle in evaluating which blog posts and podcasts are appropriate and high quality for resident education. Often residents independently access FOAM resources available online, but do not receive training credit for this. We seek to use our expertise to bridge this divide by assisting residency programs to promote asynchronous learning, online education, as well as reward residents who already effectively utilize FOAM resources.

Who is our sponsor?

The Council of EM Residency Directors (CORD), who shares our vision for improving graduate medical education through digital learning platforms, has generously offered to help sponsor our efforts. This helps toward sustaining the ALiEM AIR Series on the blog and eventually making a major move to our custom learning management platform ALiEMU (anticipated starting February-March 2016).

How do we select the articles?

The 9-person Executive Board reviews numerous FOAM resources and scores each on an objective grading system designed for this series. We curate relevant posts from top blogs and podcasts, as graded by the Social Media Index, with the inclusion criteria of publication date within the past 12 months. If this list of posts is insufficient, we broaden to a longer time interval. We do not claim an exhaustively comprehensive search, but feel that we ultimately highlight quality posts for residency education.

The cumulative highest scoring 6-10 posts from each module are selected. To decrease any obvious bias, members of the Executive Board recuse themselves from evaluating posts they wrote or were directly involved in writing. Our scoring tool looks at five measurement outcomes, each using a Likert 7-point scale:

  • BEEM Score
  • Content Accuracy
  • Educational Utility
  • Evidence Based Medicine
  • Referenced

The actual grading instrument can be viewed in this AIR Series Grading Tool PDF.

What is III?

In 2008, the Residency Review Committee (RRC) in Emergency Medicine as well as the Accreditation Council for Graduate Medical Education (ACGME) endorsed a change in the education hours requirements for residencies. Previously, five hours of educational conference time was required per week. The new recommendation allows residencies to decrease their conference time to four hours per week with an additional one hour of documented asynchronous learning outside the classroom. This series abides by the CORD and ACGME’s requirements for III. Upon request, all EM program directors will have access to our master read-only list of residents who complete the quiz. In addition, faculty on our Executive Board will engage with residents in the blog comment section for questions and comments that arise during reading.

How does this count for III credit?

Per the ACGME and EM RRC [PDF, pg 4-5], III credit depends on fulfilling all 4 of the following criteria, and this is how we address them:

  1. The program director must monitor resident participation. – We give read-access to PDs.
  2. There must be an evaluation component. – We have a post-module quiz.
  3. There must be faculty oversight. – We have faculty not only screening the content for accuracy and value, but also monitoring the blog comments section if there are questions to reply in a timely fashion.
  4. The activity must be monitored for effectiveness. – We have an internal QI process as well as a post-module survey to incorporate learner feedback.

I am an EM resident desiring III credit

As of June 2017, we now have over 125 US EM residency programs registered with over 1200 residents completing at least one module. Please speak with your program director about the possibility of your program joining the ALiEM AIR Series. If it is approved for credit at your residency, complete the end-of-module quiz and your participation will be tracked. Of course, you are always welcome to take the quiz for your own learning purposes.

I am an EM residency program interested in implementing this series for III credit

Please contact us for a log-in access to the read-only Google Drive document with all resident participants who have taken the quiz who self identify with your residency program. Please give us the email that is linked to your Google Drive account. This list is sortable by program and date. We recommend that each module of curated posts count for 2 hours of III.

Frequency of AIR series modules

We plan to release new curated lists to correspond with the EM modular curriculum.

AIR Modules

Module Date Link Recommended III
24 November 2016 Toxicology Module 2 hours
23 October 2016 Trauma Module 7 Hours
22 September 2016 Respiratory Module 4 Hours
21 August 2016 Infectious Disease Module 6 Hours
20 July 2016 Procedure 4 Hours
19 May 2016 Ortho (Lower Extremities) 1 Hour
18 April 2016 Ortho (Upper Extremities) 5 Hours
17 March 2016 Cutaneous 1 Hour
16 February 2016 Neurology 2 (headaches, Seizures, and Other) 6 Hours
15 January 2016 Neurology (bleeds and strokes) 6 hours
14 November 2015 Environmental Module 2 1 Hour
13 September 2015 Environmental Module (Bites, Temp, Stings) 1 Hour
12 August 2015 Cardiology 2 (CHF and others) 4 Hours
11 July 2015 Cardiology 1 (ACS) Module 5 Hours
10 June 2015 Respiratory 2 (PE) Module 5 Hours
09 May 2015 Respiratory 1 (General) Module 5 Hours
08 April 2015 GU/Renal Module 4 Hours
07 March 2015 Ob/Gyn Module 3 Hours
06 January 2015 Toxicology Module 4 Hours
05 December 2014 Psychiatry Module 2 Hours
04 November 2014 Peripheral Vascular Disease Module 3 Hours
03 September 2014 Endocrinology Module 3 Hours
02 August 2014 HEENT Module 2 Hours
01 July 2014 ID/Heme/Onc Module 2 Hours

Members of AIR Executive Board

  • Felix Ankel MD, Vice President and Executive Director of Health Professional Education, HealthPartners Institute for Education and Research; Associate Professor of EM; University of Minnesota
  • Jeremy Branzetti MD, Associate Program Director and Assistant Professor of EM; University of Washington Department of EM
  • Andrew Grock MD, Associate Director AIR Executive Board, EM Resident at Kings County EM Program
  • Nikita Joshi MD, Associate Director AIR Executive Board, ALiEM Associate Editor; Clinical Instructor, Stanford University, Division of EM
  • Michelle Lin MD, Associate Director AIR Executive Board, ALiEM Editor-in-Chief, UCSF Associate Professor and Academy Endowed Chair for EM Education, San Francisco General Hospital
  • Eric Morley MD, Associate Director AIR Executive Board, Associate Program Director of EM; Stony Brook University
  • Anand Swaminathan MD MPH, Assistant Program Director and Assistant Professor of EM, NYU Medical Center; Faculty Reviewer of EMLyceum.com
  • Taku Taira MD, Associate Program Director and Assistant Professor of EM, USC School of Medicine
  • Lalena Yarris MD MCR, Program Director and Associate Professor of EM; Oregon Health Sciences University

Additional Reading

  1. Mallin M, Schlein S, Doctor S, Stroud S, Dawson M, Fix M. A Survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014 Apr;89(4):598-601. PMID: 24556776.
  2. Reiter DA, Lakoff DJ, Trueger NS, Shah KH. Individual interactive instruction: an innovative enhancement to resident education. Ann Emerg Med. 2013 Jan;61(1):110-3. PMID: 25520994.

Updated 7/25/14: Added the ACGME/RRC criteria for III credit.


Andrew Grock, MD

Andrew Grock, MD

Lead Editor/Co-Founder of ALiEM Approved Instructional Resources (AIR)
Assistant Professor of Emergency Medicine
UCLA Emergency Medicine Department
Andrew Grock, MD

Latest posts by Andrew Grock, MD (see all)

  • Timothy C Peck, MD

    BIDMC is considering adopting AIR. Has there been a confirmation/release from the ACGME that you may point us to?

    • Michelle Lin, MD

      Good question, Tim. Glad to hear that BIDMC is considering this. We wanted to do this from a more rigorous perspective with our 5-item scoring tool used by our 9-person editorial team. We had run this by several program directors to see if this fulfills the III criteria and they all felt that it did. The 4 criteria are:

      1. The program director must monitor resident participation. – We give read-access to PDs.
      2. There must be an evaluation component. – We have a post-module quiz.
      3. There must be faculty oversight. – We have faculty monitoring the blog comments section if there are questions and reply in a timely fashion (like right now!).
      4. The activity must be monitored for effectiveness. – We have an internal QI process as well as a post-module survey to incorporate learner feedback.

      The III options that are used at various programs that I know of haven’t asked for ACGME “approval”. That being said, your question has sparked some interesting discussions. Will get back to you.

      Attaching the ACGME EM RRC guidelines and the specific relevant area. (https://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/110_emergency_medicine_FAQs_07012013.pdf)

      • Jessica

        Love to see collated/legitimized FOAM out there for the medical community. However, the ACGME FAQs are pretty explicit that answering questions/a quiz doesn’t count as individualized interactive instruction: “It is important to note that simply reading or answering questions does not meet the requirements for planned educational activities.” The ACGME notes that there must be real time interactive discussion/real-time questions and answers. How is a blog meeting this criteria, since not everyone is online simultaneously and interacting? Comments after the fact and blog posts seem to fall under the category of passive learning. Ie it’s great information, but where’s the live interaction/timely feedback that goes with the quiz? I’d hate to see the ACGME deny this as a valuable resource, but I agree we need confirmation that this fits the bill before we put all our eggs in this basket.

        • Michelle Lin

          Hi Jessica: Thanks for sharing these honest opinions. I’ve been reading and re-reading the criteria (page 4-5 of https://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/110_emergency_medicine_FAQs_07012013.pdf specifically).

          I agree that “It is important to note that simply reading or answering questions does not meet the requirements for planned educational activities.” Simply reading Rosen’s or taking PEER questions alone shouldn’t count. The guidelines then go on to explain that you need to instead fulfill all 4 listed criteria:

          1. The program director must monitor resident participation.
          2. There must be an evaluation component.
          3. There must be faculty oversight.
          4. The activity must be monitored for effectiveness.

          I think the issue is — what is “faculty oversight”? I can’t seem to find the wording about “real time interactive discussion/real-time questions and answers” needed by faculty. I wonder if it’s people’s interpretation of “faculty oversight”? It’s never quite defined. I wonder if having our national faculty group available for feedback is sufficient. While it’s not live, we are matching the learning modality. The education is asynchronous, so what can’t the faculty feedback be asynchronous? We do plan to give feedback within 24-28 hours. This is possible of automatic email notifications. (For instance, I was emailed your comment and am replying within 1 hour.)

          Could you forward me where you find the real-time feedback criteria? Need help finding this. Thanks! Hope the AIR Series fulfills all the 4 criteria.

  • Jordan, MD

    This is fantastic, just got word that Carolinas (where i just started my US fellowship) is on board. Love this. Great for the future of EM. We all know, much learning is done the way we all do almost everything now — web-based.

  • Paul Jones

    This is a promising development!

  • emresblog

    Cool idea. I had been toying with doing the same thing for my blog, i.e. providing a quiz that reinforces the concepts of each podcast, has an explanation for answers (providing real time feedback). Many of the quiz/test making software I found allows you to have feedback for right/wrong answers, and you can email test takers a certificate of completion and they can even download the test/explanation. Again, if you solicit feedback, you hit point number 4, and myself in my case (or your AIR team in yours) and the PD’s (who can review the content for accuracy) should count as faculty oversight, especially if you monitor your comments section closely. Agreed that I don’t know of people getting permission for all of this specifically from the ACGME. It would seem you have hit al the criteria for III, and it would then be up to PD’s to yea or nay it for their specific program. You have a pretty rigorous process, so would imagine this would work. Great idea, and hopefully residencies start to jump on this. Interesting for people in my shoes who want to provide it, but not sure what the buy in will be.

    • Yup, thanks for your support! I think you hit on a point that we already want to improve on for the future — recording their wrong answers, and providing individualized feedback/explanation based on their answers. We did this since our initial goal was to lower the barriers to implementation–

      • It’s free.

      • ALiEM is shouldering most of the administrative burden

      • We used Google Forms to collect the data (free system!), but we didn’t want to get bogged down in writing complex response-based explanations yet. We just wanted to launch the series. It’s too good to delay its release because it lacks fancy features. Innovation is an iterative process.

      • We created a logic-based quiz whereby you have to answer the right answer before progressing forward, partly because all program directors have access to the master spreadsheet. So the master sheet just provides a list of residents who completed the quiz (and by definition got a 100% score). We didn’t want any “personal” information to accidentally get out, despite the sheet being password protected (e.g. performance on exam).

      Thanks for sharing your experience/ thoughts with your podcast. We’re on the same page.

  • Sarah Kuehnle, MD

    Really enjoyed my first AIR experience! Did the ID and hem/onc one this evening. How often do you think you guys will be posting new ones? Working on seeing if this is something that would be possible at our Maricopa EM residency. Thanks again for the resources!