EM Match Advice: Program Directors Reflect on the 2021 Residency Match

How competitive is EM emergency medicine match EM Match Advice

Dr. Mike Gisondi and Dr. Michelle Lin return for their annual review of the most recent Match in emergency medicine (EM) in latest episode of EM Match Advice. They were joined by a panel of 3 outstanding program directors, Dr. Jacob Ufberg (Temple), Dr. Amita Sudhir (University of Virgina), and Dr. William Caputo (Staten Island Medical Center). Was the EM Match more competitive this year? That’s a complicated question and you need to listen to the podcast discussion for the nuanced answer.

The Slide: The Nitty Gritty Annual Numbers of the EM Match

“The Slide” above is a summary of EM Match statistics taken from the annual National Residency Matching Program (NRMP) Data and Results publication over the last 10 years. Trends suggest that EM is becoming less competitive year over year; however, 2021 saw the greatest number of unmatched medical students who were EM bound. What are the most competitive specialties and how does EM compare? Generally, we use the % fill rate with LCME students to determine the competitiveness of a specialty. That is the % of entry-level positions filled by allopathic senior medical students from the US, Puerto Rico, and Canada. Who’s on top this year? Four combined programs that each have very few PGY-1 positions in the match all filled with 100% LCME students: EM-anesthesiology, medicine-anesthesiology, pediatrics-anesthesiology, and pediatrics-PMR. Here’s how a few of the other specialties fared:

  • Thoracic surgery 93.5%
  • Plastic surgery 89.3%
  • Vascular surgery 82.3%
  • Orthopedic surgery 80.5%
  • Obstetrics-gynecology 66.7%
  • Surgery 65.6%
  • Emergency medicine 62.1%
  • Pediatrics 60.3%
  • Radiology 58.2%
  • Internal medicine 39%

Podcast: Program Directors Reflecting on the 2021 EM Residency Match

Read and Listen to the other EM Match Advice Episodes

Blog posts: https://www.aliem.com/em-match-advice-series/

By |2021-07-19T19:27:46-07:00Jul 21, 2021|EM Match Advice, Podcasts|

Education Theory Made Practical: Listen to the New Podcast Series

education theory made practical books into podcast

The Education Theory Made Practical (ETMP) set of 3 e-books were published starting 2017, reviewing key education theory principles and practically framing the abstract into practical scenarios. This series was produced by the ALiEM Faculty Incubator in collaboration with the  International Clinician Educator (ICE) blog. These e-books can be downloaded for free in the ALiEM Library.

We were surprised, humbled, and honored that Dr. Daniel Harper wanted to convert the popular e-books into a podcast series, with each chapter turned into short 10-20 minute podcast. As a pilot test, he converted Volume 1 into a set of 10 podcasts. Take a listen, put on your educator-scholar caps, and let us know what you think.

Podcast Series: Education Theory Made Practical (Volume 1)

Podcast Team

Daniel Harper

Host: Daniel Harper, MD

Senior Resident
Dual Interventional Radiology and Diagnostic Radiology
Ochsner LSU Health Shreveport

Host: Surbhi Raichandani, MD

Senior Resident
Department of Radiology
University of Arkansas Medical Sciences

Guest Voice:

  • Loren James Perley (electrical engineer)


Chan TC, Gottlieb M, Sherbino J, Boysen-Osborn M, Papanagnou D, Yarris L. Education Theory Made Practical, Volume 1. San Francisco, CA: ALiEM Publishing, 2017. [ISBN 978-0-9992825-0-2, PDF]

By |2021-07-20T06:21:16-07:00Jul 16, 2021|Academic, Medical Education, Podcasts|

How I Work Smarter: Gus M. Garmel, MD, FACEP, FAAEM

gus garmel how i work smarter

One word that best describes how you work?


Current mobile device




What is something you are working on now?

Multiple projects, presently Microaggressions & Civility in the Workplace, Communication and Success in EM, and Coaching/Mentoring in EM.

How did you come up with this Idea/Project?

These are important topics; not a lot of information is available about these topics related specifically to EM despite the need.

What’s your office workspace setup like?

Standing wrap-around adjustable desk with good lighting, multiple computer monitors, and sufficient space to work so that I can keep needed materials close and accessible in my work area. I have few distractions in my workspace, which allows me to focus best on the work I am doing.

What’s your best time-saving tip in the office or home?

Limit distractions and work on one thing at a time, which reduces inefficiencies and errors that often occur with multitasking.

What’s your best time-saving tip regarding email management?

I have several tips, although I have found that turning off email notifications and checking email infrequently (or on YOUR schedule when time allows) are perhaps the best recommendations I can share (again, this relates to multitasking inefficiencies, limiting distractions, and error prevention).

What apps do you use to keep yourself organized?

Ical, Notes, and email all help me stay organized. I also use SUPER-STICKY Post-It notes. They come in a variety of colors if you purchase them in bulk, which some people use to help with organization through color-coding (I don’t use this strategy, but it is a good one).

How do you stay up to date with resources?

Staying current and updated (medical and non-medical) is challenging and takes time. I have a few key websites bookmarked, and still get some materials through the mail on paper. I schedule time for keeping up. Some aggregated links direct me to articles of interest, and I receive TOCs directly from society journals (EM and non-EM). I make a conscious effort to keep up, and spend very little (or no) time on Facebook, Twitter, and other social media platforms.

What’s your best time-saving tip in the ED?

Always think DISPOSITION (every patient needs one). I teach that if you don’t have a good idea about a patient’s disposition, you should ask more targeted questions and do a better physical examination before leaving the room. I recommend planning for test results that can only be normal, abnormal, or indeterminate. Imagine what you would do for (and with) each patient if the test results are all negative (or normal). Have a plan for indeterminate results, for positive findings, or what to do if there is a worsening in the clinical course (including persisting pain, dizziness, vomiting, shortness of breath, etc.). I also think and teach to consider what information is necessary before it is appropriate to call a consultant that I or the patient needs anyway. Often consultants appreciate hearing about a patient “early” even before all the results return (especially if it is near the end of their day while they are still in the hospital).

ED charting: Macros or no macros?

Macros, but only after I see the patient and with attention to modifying the EMR as necessary. I am meticulous about adding detail and removing anything that is incorrect from the Macro. I never use Macros in my free-text HPI.


  • What’s the best advice you’ve ever received about work, life, or being efficient?

    FOCUS – that’s key. Make every minute (or moment) count. Whenever possible, try to “finish” one task before starting another, which keeps your “to do” list as short as possible and prevents errors and inefficiencies related to multitasking or task switching.

  • What advice would you give other doctors who want to get started, or who are just starting out?

My best and most frequent advice to all physicians (especially new physicians) is to work hard (and smart), be a team player at all times, show compassion and demonstrate empathy as often as possible (always is best), and strive to improve your communication and professionalism skills. Clinical knowledge is expected. Your efficiency will improve with experience and with practice. Be kind to as many people as you can as often as you can. These are important strategies for professional success, patient satisfaction, and personal wellness.

  • Is there anything else you’d like to add that might be interesting to readers?

Enjoy your career in EM, which will be challenging yet extremely rewarding. Strive to achieve Joy and Meaning in Medicine by working with purpose. Use people’s names frequently and correctly (patients, staff, consultants, colleagues), and express genuine interest in them as people and professionals. Learning something personal about your patients and colleagues (in and outside of EM) is a sign of respect. Expressing gratitude and saying “thank you” with sincerity are always beneficial.

Who would you love for us to track down to answer these same questions?

Anyone who has demonstrated consistent long-term success in EM, and is able to share his or her successes, failures, and strategies in a clear manner.

Read other How I Work Smarter posts, sharing efficiency tips and life advice.

How I Work Smarter: Christopher Lloyd, DO

One word that best describes how you work?


Current mobile device

iPhone 12 Pro


iPhone Xr

What is something you are working on now?

Qualitative analysis on resident perceptions of feedback

How did you come up with this Idea/Project?

As a program we are continuing to look at how feedback is being delivered to residents, when it is happening, how it is received/implemented, etc. This project grew from a desire to explore the resident perspective on these topics so as to understand better where we are effective with our feedback techniques and practices and where we can find areas to improve.

What’s your office workspace setup like?

Currently I’m sitting on my back patio while my three kids are across the yard in their hammock cocoons. I’ve never been much of a desk person, and I always am more comfortable and productive when I vary my environment. Kitchen table, living room by the window, backyard, or, preferentially, a local coffee shop – although that’s less frequent recently #thanksCOVID. Really the only constant to my workspace is my computer and a cup of coffee. Occasionally just a notepad and the cup of coffee. Always the coffee though.

What’s your best time-saving tip in the office or home?

Don’t wait until your sitting down for intentional work to start a to do list! You’ll spend the first 30 minutes of what could be some productive time thinking about what you should be doing. Find a method that works for you to collect your tasks (I’ve used a bunch…topic for another day) and have a plan for when you start to work. I mentioned working opportunistically above. There is no set schedule in our house. Both my wife and I are emergency physicians. I usually look at the week ahead and pick out where my blocks of work are going to be and then look at my task manager and pick out what/where I want to accomplish anything. Its rare that I sit down with an hour or two to work and don’t already have a plan. Second part of that is matching task management with energy levels. Say its 9am after two straight 5pm – whenever shifts (you know the one….the shift that technically has an end time but you never leave at that time). I know that after two nights of crummy sleep that I shouldn’t be trying to do any deep focused work so I’ll plan on doing lighter tasks that are quick and require less concentration. Save the stuff that takes more time and focus for days that you know you’ll be working with a full cup:) I tend to label these with either squirrel or zombie (Some call this the ‘mind is mush’ mindset) tags on my task manager so I can get a quick filter of either one depending on how I’m feeling.

What’s your best time-saving tip regarding email management?

Check it twice a day at the most. This is hard, and I fail regularly, but email is the single biggest time sink we have and the more time you spend out of email the better. The argument I hear is ‘what if it’s something important from my chair/medical director/PD?!’ Here’s the thing…if you only check it twice a day you set that expectation for others. Thankfully those people in my life know that if something is mission critical/needs addressed now they’ll call/text. Set aside this time once or twice a day, reply to the stuff that only takes 2 min or less, and add the other stuff to your task manager. I’m an inbox zero person, but I know that’s not for everyone.

What apps do you use to keep yourself organized?

I use Todoist for task management. That’s really it.

How do you stay up to date with resources?

Feedly is a RSS feed that I use to capture articles. I try to keep up to date with EMRAP and EMA because I know the residents are in that space regularly and I want to be able to speak to the topics that is on the forefront of their minds.

What’s your best time-saving tip in the ED?

Unless someone requires a life saving intervention never get more than 2 charts behind. What will take me a minute or two to dictate now will take 2-3 times that after my shift or later on and it adds up fast. Also dragon dictation. If my dragon is broken you will find me curled up in the fetal position under the desk.

ED charting: Macros or no macros?

Macros for physical exam – but make sure its your typical physical exam so you don’t need to change it often. Other wise dragon dictation for everything else. Not a fan of macros for medical decision making documentation. Too many times its obvious that its a macro and as such starts to diminish the credibility of the note.


  • What’s the best advice you’ve ever received about work, life, or being efficient?

    Work life balance implies that you have to give up one side to balance the other. I preach and practice work life integration.

  • What advice would you give other doctors who want to get started, or who are just starting out?

Sit down with patients and listen to them. You’ll save more time here than at your desk charting. Find what’s important to you and intentionally make time for that. Wellness is different for everyone.

Read other How I Work Smarter posts, sharing efficiency tips and life advice.

How We Have Kept and Will Continue to Keep Going | Summary of The Leader’s Library Discussion

keep going book

Last month, 30 people from 4 different countries and 15 different states + Puerto Rico, ranging from their early twenties to mid sixties, bravely jumped into a two-and-a-half day conversation on Slack about creativity, resilience, and their careers– the fifth version of The Leader’s Library. We discussed the book Keep Going: Ten Ways to Stay Creative in Good Times and Bad by Austin Kleon, and reflected together on the evolution of our creativity from childhood through middle age, and how a career in emergency medicine requires creativity in every possible form. Please see our earlier post for a more detailed summary of the book; below, we share the highlights of the group’s discussion.

Day 1

The first day started out with a personalized welcome video from Austin Kleon himself and introductions of participants. People shared what drew them to The Leader’s Library (connection, reflection, building community) and where they would like to incorporate more creativity (heal personal and societal wounds, bring innovative approaches to “wicked problems”, appreciate the beauty and joy in connections between things.) We then discussed the influence of specific individuals on personal creativity and how some have created a personal creative network (PCN) similar to personal learning networks (PLN). This was followed by a discussion about the spaces where people felt most creative, with many describing the benefit of being outside and disconnected from the activities of daily electronic living, while others felt most creative when connected with others through electronic means. Participants then volunteered their preferred medium for expressing creativity with lots of sharing of paintings, photography, poetry, and welding projects.

Day 2

On the second day, we covered the first 8 tenets of the book. 

1. Every day is groundhog day.

This is really about having a daily discipline: take one day at a time, establish a daily routine, and have reflective practice. Participants discussed morning and evening routines, journaling as an anchor, and the importance of routine in setting boundaries between “doing” and “being”. Some discussed an unease with routines, as our chosen lives as emergency physicians are by nature unpredictable. However, this discomfort belied participants’ flexibility and resiliency– even in pure chaos when nothing is going as planned, we’ll get through, the day/shift/week will inevitably end, and we’ll start again tomorrow.

2. Build a bliss station.

This is a “space” or “time” to disconnect from the outside world to connect with yourself. The concept of being on “airplane mode” even when not on an airplane resonated with many. Participants shared examples of their physical (outdoors, home office, kitchen island) and temporal bliss stations; for some, their bliss station was simply an extra in-tune state of mind. The conversation also revolved around the art of “saying no” and of intentionality when creating one’s career journey. Some highlights: recognize that one’s capacity is finite, and in order to say “yes” to one opportunity, one must say “no” to another; when invited to do something, ask, “What is this person really trying to achieve? Can I help them in a different way to achieve this goal?”; and journal in the days following activities and review how you felt afterward– did this project invigorate you, or were you entirely drained? Use this insight to inform future decisions.

3. Forget the noun, do the verb.

We often define our identity with who we are rather than what we do. Kleon suggests that “creative” is not a noun and that real work is play. We had a robust discussion on what “verb”-ing looks like for each of us. In looking at our careers, many of us recognized our professional “nouns” in one bucket (emails, meetings, academic rat race), and our liberating, expansive doctor “verbs” in another bucket (to help, think, read, teach). We discussed ways to contract bucket 1 (wait 24 hours to answer an email, skip meetings that aren’t action-oriented) and expand bucket 2 (step out of academia, work only on passion projects [that whole “saying no” thing again!]). 

4. Make gifts.

This was about the importance and joy of gifts. This was also about the trappings of “suckcess” and the tyranny of metrics. Kleon suggests that we should leave money on the table, we should forget to take things to the next level, and let low hanging fruit fall off and rot. Instead of the quantitative, focus on the qualitative. Be kind, be generous, be unique.

5. The ordinary + extra attention = the extraordinary.

This tenet resonated with many. Participants discussed ways they’ve cultivated over the past year to slow down and pay attention to the world around them with mindfulness techniques. With slowing down, we can finally focus attention on what we’re paying attention to, then with intentionality nurture this by giving extra attention and create something extraordinary. An interesting angle discussed how Peleton pandemic buys helped people get into the mode to conserve cognitive load and emotional labor, to slow the mind through exercise.

6. Slay the art monsters.

Kleon’s “art monsters” are those ubiquitous beings who somehow create beautiful work while behaving badly and contributing net negativity to the world. We reflected that we don’t want to become “monsters” in the same way, prioritizing output and the final project over the craft of medicine. Kleon argues here that “art is for life, not the other way around,” and this was a good reminder that we chose our careers because we wanted to make our, and others’, lives better, not to drag ourselves or others down. Many discussed this juxtaposition of simultaneously loving their practice (caring for patients, educating others) with living periods of time where pursuing their craft made themselves and others miserable. We all can have monsters coming to visit; the key is keep them around for the shortest time possible.

7. You are allowed to change your mind.

This tenet challenges the obsession with being right (hello, medicine!). Our discussion revolved around history-as-educator (“history may not repeat, but it sure does rhyme”) and philosophy-as-educator [Daily Stoic]. Some participants regularly revisit their own history through re-reading old journals; others learn through reading others’ histories in books, applying lessons from our ancestors’ missteps to our own current leadership challenges. There was also discussion of the (inverse) relationship between confidence in a position and being right. See the Dunning-Kruger prayer. In Kleon’s words, “to change is to be alive.”

8. When in doubt, tidy up.

We closed the day with a discussion of tidying our workspaces, both mental and physical. Sleep tidies up your brain. Tidying up your workspace is an exploration and a great way to focus energy when stuck or overwhelmed. Participants discussed their approach to tidying up offices, kitchens and gardens and the positive creative effects it had on their creativity. 

9. Demons hate fresh air.

On the third and final day of discussion, we covered Kleon’s final 2 tenets. This is one of the more intuitive tenets– that getting one’s bootie off one’s chair and going outside can stimulate creativity and launch us past writer’s block. However, Kleon takes this beyond endorphins and argues that, by going outside, we better integrate ourselves into our communities and reality: “If we do not get outside, if we do not take a walk out in the fresh air, we do not see our everyday world for what it really is, and we have no vision of our own with which to combat disinformation.” Participants had varied strategies for getting themselves OUT, though most agreed that getting outside had been critical to their physical and mental wellbeing over the past year. Several people cultivated a habit of photographing their surroundings while out walking, a practice which helped them stay present. One participant shared that for them, this tenet took on a metaphorical meaning– by letting their internalized shame out to “fresh air” through writing or speaking, its power over them lessened. 

10. Plant your garden.

Kleon, and our participants, focused on seasonality here. We don’t expect plants to flower in the winter, nor do we furiously cling to an oak tree’s leaves when they tumble off in the autumn; why, then, in academic emergency medicine, in medical education, in medicine in general, do we completely disregard this natural rhythm and instead attempt to overpower it with a decades-long continuous stream of hard work? Why are we surprised that our creativity stalls periodically, when we know innately that everything in our world is cyclical? Participants brainstormed ways to integrate periods of recuperation and recharge (what Kleon calls “dormancy” in this essay) into their professional lives, not just for rest, but also to enhance creative output. One participant reflected, “I’ve personally struggled with having to ALWAYS be ‘on’ in terms of being creative, but now understand that creativity ebbs and flows and the key now is to capitalize when the time is right.” 

Synchronous Conversation

We closed out The Leader’s Library with a live video conversation, during which participants ranging from medical student to late-career professor mused about the connections they formed during The Leader’s Library, new ideas they’d be taking back to their institutions, and shocking realizations they’d had while learning from and with their co-participants. Independent of differences in prior life experiences and current situations, all participants affirmed a renewed appreciation for the role of creativity in their professional lives.


Whether in the US or India, academic or community emergency medicine, medical student or faculty, leading or trying to lay low, our facilitators and participants all need to keep going– and the past year plus has made this an incredible challenge. We’ve known a solution to this challenge since our preverbal years– taking a handful of crayons to a blank page helped us cope with doctor’s visits and tortuously slow restaurant service, art allowing our brains to take a break from the world around us and observe with a new lens. Participants left the fifth iteration of The Leader’s Library with plans to better support creativity in themselves and their colleagues, with a goal of improving not only the quality of their work, but also the quality of their existences

The facilitation leadership team were wholly inspired by the participants and their vulnerability and candid insight. Stay tuned for our next turn in the fall and until then, ta-ta for now!

By |2021-06-17T10:47:44-07:00Jun 29, 2021|Book Club, Leaders Library, Medical Education|

GroundED in EM: A new ALiEMU course series for third-year medical students

GroundED in EM curriculum medical student

During the pandemic, similar to how a work-from-home mentality has become more accepted, a learn-on-own mentality has arisen for medical students. The success of the 9-part Bridge to Emergency Medicine (EM) self-guided curriculum for senior medical students interested in EM has confirmed this. This was evidenced by over 130,000 page views about the Bridge curriculum since March 2020 and 609 awarded ALiEMU certificates since April 2021 (launched only 2 months ago!).

GroundED in EM: A new curriculum for third-year medical students

We are thrilled to announce a 4-week, self-guided reading/listening curriculum along with choose-your-own-adventure cases paired now with ALiEMU quizzes, certificates, and badges for third-year medical students interested in EM. It’s called GroundED in EM, and created by an all-star team led by GroundED Editor-in-Chief, Dr. Andy Little. Here’s the rest of the team:


  • Brian Barbas, MD, FACEP (Associate Professor of Emergency Medicine, Loyola University Chicago – Stritch School of Medicine)
  • Carmen J. Martinez, MD MSMEd (Assistant Professor of Emergency Medicine, University of South Alabama)
  • Guy Carmelli, MD, MSEd (Assistant Professor of Emergency Medicine, University of Massachusetts)
  • Laryssa A. Patti, MD (Assistant Professor of Emergency Medicine, Rutgers Robert Wood Johnson Medical School)

Adventure Co-Creators:

  • Kaitlin Bowers, DO (Vice Chair of Emergency Medicine, Campbell University College of Osteopathic Medicine)
  • Meenal Sharkey, MD FACEP (Assistant Program Director & Clerkship Director; Department of Emergency Medicine, Doctors Hospital)

GroundED on ALiEMU

Similar to Bridge to EM, reading and listening materials have been identified and curated from external sites. Then come on back to ALiEMU to take self-assessment quizzes to get your certificates and badges.

By |2021-06-22T13:20:35-07:00Jun 22, 2021|ALiEMU, Medical Student|

Are You Using the Right Evaluation Tool to Assess Learners? Putting Validity on Trial

Evaluation Tool for Learners - Validity on Trial

As medical educators, we often rely on assessment tools to evaluate our learners. Whether in the form of a post-lecture survey or a checklist by a standardized patient assessment, tools are used throughout medical training and beyond. How do we know the tool we are using is appropriate? Is it assessing the right things? Do the scores have any meaning? We often search for tools that have been “validated” and feel more confident applying the results. But what makes a tool “valid”? With a few simple concepts, we can better choose and create our assessment tools and therefore better cater our education to the needs of learners. 

What is Validity?

Defining validity is challenging, especially when the terms seem to be redefined just as soon as we get comfortable with them. Even before validity, we must address the construct (what the instrument is intended to measure). Most often this construct does not have a standard or inherent normal or abnormal, such as physician attitudes or patient symptoms. Messick defines construct as “an intangible collection of abstract concepts inferred from behavior and used to measure validity”. [1]  Defining the construct is the first step in creating an assessment tool or choosing an appropriate pre-conceived tool. 

When applied to assessments, validity is a hypothesis not a statement of fact. A hypothesis requires evidence, either in support or opposition. A tool itself is not “valid” or “invalid” but instead the interpretation of whether the data has or does not have validity [2]. A tool may have validity in one context but not another; with one type of learner but not another. Dr. David Cook, an expert in medical education validation research, defines validity as “the degree to which the interpretations of scores resulting from an assessment activity are ‘well-grounded or justifiable’” [3].

The Courtroom Analogy

Validity lies on a continuum and relies on 4 foundational concepts:

  1. Propositions
  2. Evidence
  3. Argument
  4. Decisions

Cook uses the analogy of the courtroom to simplify these concepts [3].


Start with the prosecution, who proposes that the defendant is guilty. This proposition is the basis of the trial and evidence will be presented to support this proposition. Similarly, propositions guide the collection of validity evidence and are essential when evaluating the validity of a tool. Propositions are to the validity hypothesis as objectives are to the goal. For example, I propose my leadership assessment tool will include elements identified as essential to effective leadership in a resuscitation. 


Next, the prosecution presents its evidence or a collection of evidence. One eyewitness does not make a case. However, an eyewitness, DNA evidence, and a motive might seal the deal. In the assessment of validity there are 5 main types of evidence, defined by Messick [1, 2].

  1. Content evidence asks whether the instrument completely represents the construct. To return to the leadership assessment tool example, does the tool truly measure leadership skills? Are the questions, or items, important and necessary? Are there too many or too few? 
  2. Internal structure refers to reliability of the instrument, including interrater-reliability and test-retest reliability. Is score variation among participants expected? We would expect novice leaders to score lower than seasoned attendings. Are scores from different observers similar? 
  3. Response process refers to the relationship between the intended construct and the thought process of the subjects or observers. Do those being assessed understand the items on the tool as intended? If not, the tool is not assessing your construct as expected. 
  4. Consequences, intended and unintended, of an assessment can affect the tool’s validity. Do low scores lead to remediation and therefore improved performance? Or alternatively, do low scores cause self-doubt and decreased confidence, leading to poor performance? 
  5. Relation to other variables, previously known as construct validity, refers to the correlation of scores to other tools that assess the same construct. How does my tool compare to other leadership assessment tools? Just like in a courtroom, more corroborating evidence is better, but you don’t need evidence in every category to get a conviction or acquittal. 


After collecting the evidence, each side has an opportunity to make their arguments. As Cook states “the evidence doesn’t speak for itself” and a strong validity argument requires the structured presentation of evidence [3].


How will the tool be used and what is the effect of its use? How much evidence is necessary to use the instrument in a certain environment or with a group of learners depends on how the scores will be interpreted. Educational assessments of learners used primarily for the educator’s benefit to develop a curriculum are arguably low stakes and require less evidence before use. However, assessment tools with significant long-term consequences, such as remediation or a failing grade, are high stakes and require stronger validity evidence prior to application [3].

How to Choose a Tool

When assessing an evaluation tool to be used with learners, consider the above concepts of validity and specifically the validity evidence provided. A tool with presented evidence in multiple categories and with the plan to be used on a similar population and/or environment is ideal, although not always possible. Creating your own tool may be necessary [4]. When doing this, consider testing the instrument’s validity before applying the tool. And then consider the potential outcome of the application of the tool and its significance to the learner. A high-stakes outcome, such as pass/fail or granting of increased autonomy, requires assessment tools with large amounts of validity evidence to be applied with confidence. Unfortunately, in medical education we are guilty of using assessment tools frequently that do not meet these standards. If we pause and ask some basic questions about our instrument and what it is assessing we can better choose and create tools that truly have benefit to our learners. 


  1. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-104). New York, NY: American Council on education and Macmillan.
  2. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119(2): doi:10.1016/j.amjmed.2005.10.036. PMID: 16443422
  3. Cook DA. When I say… validity. Med Educ. 2014;48(10):948-949. doi:10.1111/medu.12401. PMID: 25200015
  4. Reid J, Stone K, Brown J, et al. The Simulation Team Assessment Tool (STAT): development, reliability and validation. Resuscitation. 2012;83(7):879-886. doi:10.1016/j.resuscitation.2011.12.012. PMID: 22198422

Additional Reading

  • ALiEM Education Theories Made Practice eBooks [ALiEM Library]
  • Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830-837. doi:10.1046/j.1365-2923.2003.01594.x PMID: 14506816
  • Zamanzadeh V, Ghahramanian A, Rassouli M, Abbaszadeh A, Alavi-Majd H, Nikanfar AR. Design and Implementation Content Validity Study: Development of an instrument for measuring Patient-Centered Communication. J Caring Sci. 2015;4(2):165-178. Published 2015 Jun 1. doi:10.15171/jcs.2015.017. PMID: 26161370
  • Kessler CS, Kalapurayil PS, Yudkowsky R, Schwartz A. Validity evidence for a new checklist evaluating consultations, the 5Cs model. Acad Med. 2012;87(10):1408-1412. doi:10.1097/ACM.0b013e3182677944, PMID: 22914527
  • Ayre C, Sally AJ. Critical Values for Lawshe’s content validity ratio: revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development. 2014;47(1),79-86. doi:10.1177/0748175613513808

By |2021-06-16T09:29:35-07:00Jun 19, 2021|Medical Education|
Go to Top