Are You Using the Right Evaluation Tool to Assess Learners? Putting Validity on Trial

Evaluation Tool for Learners - Validity on Trial

As medical educators, we often rely on assessment tools to evaluate our learners. Whether in the form of a post-lecture survey or a checklist by a standardized patient assessment, tools are used throughout medical training and beyond. How do we know the tool we are using is appropriate? Is it assessing the right things? Do the scores have any meaning? We often search for tools that have been “validated” and feel more confident applying the results. But what makes a tool “valid”? With a few simple concepts, we can better choose and create our assessment tools and therefore better cater our education to the needs of learners. 

What is Validity?

Defining validity is challenging, especially when the terms seem to be redefined just as soon as we get comfortable with them. Even before validity, we must address the construct (what the instrument is intended to measure). Most often this construct does not have a standard or inherent normal or abnormal, such as physician attitudes or patient symptoms. Messick defines construct as “an intangible collection of abstract concepts inferred from behavior and used to measure validity”. [1]  Defining the construct is the first step in creating an assessment tool or choosing an appropriate pre-conceived tool. 

When applied to assessments, validity is a hypothesis not a statement of fact. A hypothesis requires evidence, either in support or opposition. A tool itself is not “valid” or “invalid” but instead the interpretation of whether the data has or does not have validity [2]. A tool may have validity in one context but not another; with one type of learner but not another. Dr. David Cook, an expert in medical education validation research, defines validity as “the degree to which the interpretations of scores resulting from an assessment activity are ‘well-grounded or justifiable’” [3].

The Courtroom Analogy

Validity lies on a continuum and relies on 4 foundational concepts:

  1. Propositions
  2. Evidence
  3. Argument
  4. Decisions

Cook uses the analogy of the courtroom to simplify these concepts [3].

Propositions

Start with the prosecution, who proposes that the defendant is guilty. This proposition is the basis of the trial and evidence will be presented to support this proposition. Similarly, propositions guide the collection of validity evidence and are essential when evaluating the validity of a tool. Propositions are to the validity hypothesis as objectives are to the goal. For example, I propose my leadership assessment tool will include elements identified as essential to effective leadership in a resuscitation. 

Evidence

Next, the prosecution presents its evidence or a collection of evidence. One eyewitness does not make a case. However, an eyewitness, DNA evidence, and a motive might seal the deal. In the assessment of validity there are 5 main types of evidence, defined by Messick [1, 2].

  1. Content evidence asks whether the instrument completely represents the construct. To return to the leadership assessment tool example, does the tool truly measure leadership skills? Are the questions, or items, important and necessary? Are there too many or too few? 
  2. Internal structure refers to reliability of the instrument, including interrater-reliability and test-retest reliability. Is score variation among participants expected? We would expect novice leaders to score lower than seasoned attendings. Are scores from different observers similar? 
  3. Response process refers to the relationship between the intended construct and the thought process of the subjects or observers. Do those being assessed understand the items on the tool as intended? If not, the tool is not assessing your construct as expected. 
  4. Consequences, intended and unintended, of an assessment can affect the tool’s validity. Do low scores lead to remediation and therefore improved performance? Or alternatively, do low scores cause self-doubt and decreased confidence, leading to poor performance? 
  5. Relation to other variables, previously known as construct validity, refers to the correlation of scores to other tools that assess the same construct. How does my tool compare to other leadership assessment tools? Just like in a courtroom, more corroborating evidence is better, but you don’t need evidence in every category to get a conviction or acquittal. 

Argument

After collecting the evidence, each side has an opportunity to make their arguments. As Cook states “the evidence doesn’t speak for itself” and a strong validity argument requires the structured presentation of evidence [3].

Decision

How will the tool be used and what is the effect of its use? How much evidence is necessary to use the instrument in a certain environment or with a group of learners depends on how the scores will be interpreted. Educational assessments of learners used primarily for the educator’s benefit to develop a curriculum are arguably low stakes and require less evidence before use. However, assessment tools with significant long-term consequences, such as remediation or a failing grade, are high stakes and require stronger validity evidence prior to application [3].

How to Choose a Tool

When assessing an evaluation tool to be used with learners, consider the above concepts of validity and specifically the validity evidence provided. A tool with presented evidence in multiple categories and with the plan to be used on a similar population and/or environment is ideal, although not always possible. Creating your own tool may be necessary [4]. When doing this, consider testing the instrument’s validity before applying the tool. And then consider the potential outcome of the application of the tool and its significance to the learner. A high-stakes outcome, such as pass/fail or granting of increased autonomy, requires assessment tools with large amounts of validity evidence to be applied with confidence. Unfortunately, in medical education we are guilty of using assessment tools frequently that do not meet these standards. If we pause and ask some basic questions about our instrument and what it is assessing we can better choose and create tools that truly have benefit to our learners. 

References:

  1. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-104). New York, NY: American Council on education and Macmillan.
  2. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119(2): doi:10.1016/j.amjmed.2005.10.036. PMID: 16443422
  3. Cook DA. When I say… validity. Med Educ. 2014;48(10):948-949. doi:10.1111/medu.12401. PMID: 25200015
  4. Reid J, Stone K, Brown J, et al. The Simulation Team Assessment Tool (STAT): development, reliability and validation. Resuscitation. 2012;83(7):879-886. doi:10.1016/j.resuscitation.2011.12.012. PMID: 22198422

Additional Reading

  • ALiEM Education Theories Made Practice eBooks [ALiEM Library]
  • Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830-837. doi:10.1046/j.1365-2923.2003.01594.x PMID: 14506816
  • Zamanzadeh V, Ghahramanian A, Rassouli M, Abbaszadeh A, Alavi-Majd H, Nikanfar AR. Design and Implementation Content Validity Study: Development of an instrument for measuring Patient-Centered Communication. J Caring Sci. 2015;4(2):165-178. Published 2015 Jun 1. doi:10.15171/jcs.2015.017. PMID: 26161370
  • Kessler CS, Kalapurayil PS, Yudkowsky R, Schwartz A. Validity evidence for a new checklist evaluating consultations, the 5Cs model. Acad Med. 2012;87(10):1408-1412. doi:10.1097/ACM.0b013e3182677944, PMID: 22914527
  • Ayre C, Sally AJ. Critical Values for Lawshe’s content validity ratio: revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development. 2014;47(1),79-86. doi:10.1177/0748175613513808

By |2021-06-16T09:29:35-07:00Jun 19, 2021|Medical Education|

How I Work Smarter: Miguel Reyes, MD

One word that best describes how you work?

Comfortable

Current mobile device

iPhone 12 Pro

Computer

Macbook Air

What is something you are working on now?

Wound Care article, REBEL EM CME content, Journal Reviews

How did you come up with this Idea/Project?

It was an opportunity that presented itself during the fellowship. It’s a collaborative effort with other faculty members to pull together this large review article, its a lot of work and effort but I think it’ll be worth it. As for the REBEL content, I’ve been working with Salim for a little bit and this chance came up so I decided I wanted to help upload the content to be made into CME.

What’s your office workspace setup like?

Well, it used to be the Kitchen Counter (NYC apartment doesn’t offer much space) but recently got a little desk and chair in our bedroom so I sometimes use that.

 

 

What’s your best time-saving tip in the office or home?

Close all distractions and put away your phone. When working on a project consider it your “deep work” time and focus your energy on that.

What’s your best time-saving tip regarding email management?

Check it only twice a day. Once in the morning and the other time in the afternoon.

What apps do you use to keep yourself organized?

Strides – My habit tracker for things I want to improve on, studying EM topics

Todoist – Great app for being able to break down large daunting projects into smaller manageable tasks while keeping it all organized. Since downloading this app I’ve become significantly more productive.

How do you stay up to date with resources?

Feedly – News aggregator website. I simply link all the FOAM sites I really like to it so when a new article comes out I’m interested in I can read it there.

What’s your best time-saving tip in the ED?

When you dispo the patient, finish the chart, and every time you stand up from your seat try to do at least 3 tasks before sitting back down.

ED charting: Macros or no macros?

Macros, otherwise I’d be charting for ages and there isn’t enough Great British Bake Off to numb that kind of pain

Advice

  • What’s the best advice you’ve ever received about work, life, or being efficient?

    Discipline equals freedom – Sounds trite but its transformed my outlook. The more disciplined I’ve become with social media and focused work time has allowed me to be more present with my family and loved ones for the time that really matters.

    Dr. Bove one of our staff at St. Joes gave us what I thought was great efficiency advice. If you wanna be good and have a good flow in the department every time you stand from your seat to do a task, do as many as possible before sitting back down.

  • What advice would you give other doctors who want to get started, or who are just starting out?

Don’t rush the outcome. The fun in this is not the destination but the journey in getting there so try to enjoy all the twists and turns along the way.

Who would you love for us to track down to answer these same questions?

  • Marco Propersi, DO

Read other How I Work Smarter posts, sharing efficiency tips and life advice.

By |2021-05-31T15:34:38-07:00Jun 2, 2021|How I Work Smarter, Medical Education|

How I Work Smarter: Laryssa Patti, MD

One word that best describes how you work?

Organized chaos (that’s 2 words)

Current mobile device

iPhone XS

Computer

2020 13″ Macbook Air + iPad with keyboard as a second monitort

What is something you are working on now?

Incorporating FOAM into our EM clerkship

How did you come up with this Idea/Project?

COVID decided for me — we’re working on a way to keep students engaged even though we’re still semi-distance learning. Additionally, I want to send my students into residency knowing that FOAM exists but needs to be assessed in the same way we think about peer reviewed literature.

What’s your office workspace setup like?

I have an office at our medical school that is a big flat surface in a square room with an internet connection. All of the art on the walls is of landscapes, no diplomas! It is off the beaten path (and also around the corner from the hospital Starbucks), which makes it a great place to get work done.

Since COVID, I’ve been doing the majority of my non-clinical work at home with essentially the same set up, but including a cat.

What’s your best time-saving tip in the office or home?

Everything goes into my Google calendar. If I need 30 minutes to read articles for our department journal club or food shopping, it goes in my calendar. It keeps me honest and on top of my to-do list. I try to protect a certain morning/afternoon a week (this year it’s Tuesday afternoon) for my own projects and writing and will defend that time from other obligations. I am still learning that I probably need 1.25-1.5x more time than I think to achieve any goal, though.

What’s your best time-saving tip regarding email management?

I am an inbox zero person. The 5 digit unread email number of one of my good friends gives me palpitations. My Gmail automatically sorts emails into buckets (e.g., advertisements, bills) so that my actual inbox is only emails that need immediate attention. If I can give a response immediately, I will. If I need more time to research an answer, I’ll reply as much, and stick it on my calendar to come back to. I’ve also really leaned into “snoozing” emails in both Gmail and Outlook. Things like the didactic conference schedule for this week or the agenda for a meeting later this week will get snoozed until 30 minutes before that time. Similarly, I schedule a lot of emails to be sent (like the medical student schedule that we send out weekly) so that I can put in a little bit of effort upfront and then trust Outlook to send that to faculty at a certain time.

What apps do you use to keep yourself organized?

Gmail, Tasks, and Google Calendar primarily. And also the notes app on my phone for random things that I don’t want to forget that come to me while driving or in the middle of the night.

How do you stay up to date with resources?

I subscribe to a few journal aggregators that send me weekly summaries of new articles and, of course, the EM:RAP and Twitter FOAM universe that sends me back to primary sources.

What’s your best time-saving tip in the ED?

Enter orders and write/dictate at least HPI/PMH after every patient. Every time I try to stack a bunch of patients (even low acuity ones) I end up getting interrupted and realize that I’m more behind than I thought.

ED charting: Macros or no macros?

Macros, with cautions. I have some standard physical exam and discharge instructions, but only use them for specific patients.

Advice

  • What’s the best advice you’ve ever received about work, life, or being efficient?

    Say no to things after your first year out. Once you know what you like, ask yourself: is the time/effort required to participate in [admissions committee/ interdepartmental project / extracurricular project] worth what you get out of it? If the answer is no, don’t do it. If the thing that you get out of it is rest/relaxation that is a valid reason.

  • What advice would you give other doctors who want to get started, or who are just starting out?

Say yes to things your first year out. This helps you meet people and learn about different niches that you can fill. But you can’t do this forever (see answer above).

  • Is there anything else you’d like to add that might be interesting to readers?

Accept that things will not go according to plan and forgive yourself. One of the best things that I did in the last month was blow off a long meeting in that I had nothing much to add besides being a member of the quorum in order to go on a walk and watch a movie with my husband. I came back to work re-energized and was much more productive the next day. Totally worth it.

In the same vein, I’m a new mom and still struggling to find a balance between being home and present, clinical work, and non-clinical work, and am continuing to remind myself that as much as my son is growing, I am growing too! (Some days work out better than others).

Who would you love for us to track down to answer these same questions?

  • Eric Blazar (@eblazar)
  • Sugeet Jagpal (@ysugeety)

Read other How I Work Smarter posts, sharing efficiency tips and life advice.

By |2021-05-17T06:44:59-07:00May 19, 2021|How I Work Smarter, Medical Education|

Reading from the Silver Linings Playbook: The ALiEM Connect Project

ALiEM Connect graduation

It feels like yesterday that we were sheltered-in-place, staring at our computers, wondering, “So now what?” 

As COVID-19 paused all in-person educational sessions, the early morning residency conference we used to begrudgingly join quickly became something that we profoundly missed. While we can now be “present” while wearing sweatpants and a button-down shirt, we miss the human connection. Many of us would gladly even suffer through traffic just to be a part of this morning conference tradition.

As educators and innovators, we know what a disruptive force the COVID-19 pandemic has been to the medical community. It has strained our medical and healthcare systems and has irrevocably altered our day-to-day lives. Without a doubt, the pandemic also changed how we delivered educational content to our learners over the past year.

Scholars have written about how likely this pandemic will likely precipitate the much-needed digital transformation of healthcare and health professions education that many of us have expected and hoped for. But while some of these innovations are born out of necessity, they may also inadvertently isolate us from the experiential aspects of education and human interaction that provide meaning to our work. For the ALiEM team, we cherish the opportunity to be part of some of these significant innovative and positive “disruptions,” further aligning our goal of creating an impactful and fulfilling academic life in emergency medicine. 

The Backstory

As a remote team working across continents, the ALiEM team has thrived on digital connection for over a decade. With excellent collaborators and volunteers representing different parts of the world, our daily operations require us to stay connected and work asynchronously to achieve our goals and deliverables. When the lockdowns hit, we leveraged its impact on physical distancing and leaned into connecting with each other even more! They say “chance favors the prepared mind,” and there we were, already on Slack and yearning for the opportunity to harness the power of teamwork using our shared passions, individual creative strengths, and enthusiastic and supportive emojis. There were moments of creating, moments of celebration, and moments of simply being with each other – often through an evening #WifiAndWine.

By the Ides of March 2020, an auspicious time indeed, we knew we were at a turning point. Our friends and work families had been working on the front lines combating the pandemic locally, gathering PPE, and studying the effects of a virus we knew next to nothing about. New information was coming in daily, and the signal-to-noise ratio was low. In some ways, to escape the disruptions going on all around us, we banded together to focus our unique energies toward creating something as novel as the virus itself in the realm of free open-access medical education.

At a time where everyone was feeling alone, we asked ourselves how we could support the joy of learning from and with each other? In truly whirlwind fashion, the first ALiEM Connect conference went from idea to execution in less than 2 weeks, a record-breaking time even for ALiEM. Thank especially to the American Board of Emergency Medicine for sponsoring these events.

We recently made it to the semi-finals at the CORD/ACEP Innovator of the Year competition, where we shared the below video capturing the fun, collaboration, and innovative outcome of our efforts. Oh, and the familiar ratatat of Slack.

Making this a Multiple Win

The secret sauce of the ALiEM team is that we have a diverse group of people, each of whom brings their own perspective and that we are able to share with one another liberally. Dr. Michelle Lin encouraged an environment that is psychologically safe and supportive since the inception of the ALiEM enterprise. It is out of this space that our diverse team was able to successfully bring a massively successful project to fruition amid a global pandemic. What started as a small brainstorming session blossomed into ALiEM Connect – 3 distinct remote conferences featuring nationally-recognized educators and thought leaders enjoyed by residents across the country.

It’s difficult to express as a linear narrative, but looking back, it seems as though our team divided into unique roles without a second thought. Just like a production company, we had the front and back of the house. Those in the front made sure to help get people in the seats to watch; stage managers and coordinators ensured that every part of each of the ALiEM Connect experiences was phenomenally smooth. We had talented individuals who acted as hosts and speakers to ensure that each of these experiences was top-notch and engaging. In the back, Drs. Mary Haas, Yusuf Yilmaz, and Teresa Chan sprung quickly into action to create a program evaluation strategy for our ALiEM Connect program, including a formal institutional review board exemption! All the while, testing and vetting platforms and methods to distribute the material were ongoing. We built upon each technological skill, learned new platforms, and trialed different features. We had barely decided on an open, free, and accessible platform (which was, in fact, no individual platform but an amalgamation of many!) before sending out the invites.

But the fun didn’t stop there! We’re the “academic” life in emergency medicine! How could we not also share our results with the traditional academic community? Within days of finishing our first ALiEM Connect experience, our program evaluation team generated the scaffolding of a manuscript to put together our thoughts and analyze the evaluation data collected. We harnessed the power of metrics from social media platforms (YouTube, Slack, Twitter), website analytics, and end-user experiences. Harnessing all of these analytics and communicating the right message with our academic medicine community was important to inform and help others to replicate similar approaches to their residents. Our team used ready to use metrics which came from YouTube analytics. But we did not stop there as we needed more reports of how the residents and programs interacted during the Connect events in the backchannel, Slack. We developed Python supported software to export and analyze all the messages happening in separate channels. We developed a “Emoji Cloud” to see how the reactions happened, and closely analyzed the messages during the event.

Given the true novelty of the experience, we figured we might as well shoot for the moon, as they say, by submitting our innovation description paper to Academic Medicine. After all, even if they didn’t accept it, we might get some constructive reviews, to say the least. As innovators, we are comfortable with the possibility of failure. We understand the value of the saying, “You miss 100% of the shots you don’t take,” and were prepared to accept “no” as an answer. With that, we took a calculated risk, making use of the same collaborative strategy to craft a manuscript, and clicked submit.

…And we’re glad we took that shot! We are excited to share that what we sent was indeed accepted and express our gratitude for the chance to share our low-cost approach to a large-scale, nationwide residency conference! You may read the Published Ahead-of-Print version of our paper.

Moral of the story…

You might be asking yourself, “What’s the moral of the story here? Of course, with enough academics and experts, yeah, you got a paper published. Cool…” But the papers aren’t the point. In fact, during the COVID-19 pandemic, more papers have been published than ever before – more research is being done, and our whole field is changing. The point is… this is how we got to ENJOY the academic life during a pandemic! We made lemonade (and several other desserts!) out of the lemons we were handed. New knowledge comes from thinking big and trying new things. Turns out, sometimes you also have to write about those experiences and share them with others.

As emergency physicians, we know we’re good in a crisis. But this experience reminded us that by surrounding ourselves with amazing people, we could get a surprising amount of work done (at record speed) and have a fantastically memorable time along the way. The moral of this story is that when you bring great people together and give them a chance to get to know each other, magic happens. ALiEM Connect happens. And we impact more people than we can possibly meet at the touch of our keyboards. We are so grateful for the chance to work alongside all the wonderful people at each of our institutions every day. Still, also, we are indebted to those who are our digital family. Thank you to all of you who make initiatives like ALiEM Connect possible. Academic life in emergency medicine is all about bringing a great team together.

So is the ALiEM team.

How I Work Smarter: Sara Dimeo, MD

One word that best describes how you work?

Collaboratively

Current mobile device

iPhone 11 Pro

Computer

MacBook Pro

What is something you are working on now?

The Impact of Digital Badges on Motivation in Asynchronous Learning

How did you come up with this Idea/Project?

When COVID hit, we had to think creatively on how to engage learners in an online format. Having done a fellowship in Multimedia, Design, Education Technology I was really excited to experiment with different techniques. My main goal is always to create a sense of engagement, even when there is limited ability to do so in a traditional way. One thing I’d seen done in a limited fashion are badges to reward learners, so I decided to explore digital badges and learned they’re becoming increasingly popular in the K-12 literature and other fields. I adopted this to create our own asynchronous curriculum with a badge system to identify learners who were engaging well in the material, and who were performing highly on knowledge-based quizzes.

What’s your office workspace setup like?

I finally have a dedicated office space in my home, which I love! I work best when I’m on my own away from distractions because I’m a typical EM personality. The window in my office looks out into our yard and the front street which is nice for a mental break. I like to keep my desktop clean (clean desk = clean brain) and just have my laptop and a notepad available to jot down quick thoughts or reminders.

What’s your best time-saving tip in the office or home?

Blocking off chunks of time to accomplish xyz task is helpful for me, as I often have multiple projects and/or tasks on any one given day. Prioritizing is important. I became pretty good at task-switching during my fellowship when I was juggling a lot of non-clinical projects.

What’s your best time-saving tip regarding email management?

I use a delete-flag-archive system, where anything not important in the future is immediately deleted, things that need close follow up are flagged, and everything else is filed into its relevant folder. I have 6-7 folders which I frequently use. One really helpful folder that you don’t always think about is for IDs/passwords!

What apps do you use to keep yourself organized?

I love the Things app for task management. It merges with my Google Calendar and allows you to set due dates for upcoming tasks as well as to documents to-do lists for bigger or more long-term projects. It is my peripheral brain! I also have a whiteboard in my office that displays my big projects. I like checking things off as a sense of accomplishment.

How do you stay up to date with resources?

Twitter is probably my main resource for connecting with colleagues about new ideas…I almost exclusively following folks in medical education. I use Journal Feed for quick synopses of review articles. And of course EMRAP.

What’s your best time-saving tip in the ED?

Try not to delay tough cognitive decision points that will affect the workup/algorithm that you go down. When you are not sure what to do, take a step back to think, call a consultant, or ask the advice of a colleague.

ED charting: Macros or no macros?

Yes! For example, I have an abscess I&D macro that contains less than 5 variables to fill out. Though, I dislike macros for more complicated encounters.

Advice

  • What’s the best advice you’ve ever received about work, life, or being efficient?

    [Great creative minds] think like artists but work like accountants. – Cal Newport, “Deep Work”.

    I personally love to think about new ideas or projects, but unfortunately that does not equate to success. Success is the ability to organize yourself to be productive. This is a work in progress for all of us!

  • What advice would you give other doctors who want to get started, or who are just starting out?

Get involved in a bigger way. Think about serving on a hospital committee or taking a leadership position. It will give you a new experience and accelerate your growth. For me, that was joining the EMRA Education committee, which ultimately led to me serving as the Director of Education for their Board of Directors. It was and has been a life changing experience.

  • Is there anything else you’d like to add that might be interesting to readers?

Find mentors that believe in you. I was told by an advisor that if I wanted to do med ed, I had to have my entire career path laid out and was highly discouraged from considering it. A two year fellowship and an (almost completed) masters later, I’m very glad I trusted my instinct and did not listen to their advice. Eventually I looked elsewhere and found mentors who were willing to help me achieve my goals.

Who would you love for us to track down to answer these same questions?

  • Kat Ogle @DrKittyKat
  • Kristy Schwartz @kaynani32
  • Zach Jarou @zachjarou
  • John Eicken @MedEd_Tech

Read other How I Work Smarter posts, sharing efficiency tips and life advice.

By |2021-04-20T09:12:55-07:00Apr 23, 2021|How I Work Smarter, Medical Education|

How I Work Smarter: Christopher Colbert, DO FACEP, FACOEP, FAAEM

One word that best describes how you work?

Practical

Current mobile device

iPhone 12

Computer

iMac Pro / Mac book Pro

What is something you are working on now?

Organizing/Planning the ACOEP’s annual Spring Seminar

How did you come up with this Idea/Project?

I have been part of the planning committee for the spring conference for 5 years.

What’s your office workspace setup like?

A well-lit room with decent speakers (huge music fan), 2 Macs side-by-side, and most importantly, a very comfy chair.

christopher colbert work smarter office 1

christopher colbert work smarter office 2

What’s your best time-saving tip in the office or home?

Keep your work in your office so that the remainder of your home is a comfortable place “away” from work. This concept will ensure that you maintain focus when completing your office work and support a healthy home that is not overrun by work responsibility outside of the office.

What’s your best time-saving tip regarding email management?

Anything by email that can be completed in 2 minutes or under, make time to complete if the moment is available. It is easy to allow the inbox to overflow throughout the course of a day. Periodically checking one’s email with purpose will ensure that the burden of finishing emails for the course of the day is no more than five minutes.

What apps do you use to keep yourself organized?

Google calendar is what I live by :)

How do you stay up to date with resources? 

FOAMed and journal watch

What’s your best time-saving tip in the ED?

Every time you sit down, document on two charts. As ED physicians we live and breathe based on documented reevaluations.  my recommendation to keep up with a busy room is to document on at least 2 charts each time.

ED charting: Macros or no macros?

Macros expedite charting and make each chart more robust

What’s the best advice you’ve ever received about work, life, or being efficient? 

The key is not to prioritize what’s on your schedule but to schedule your priorities.

– Stephen Covey

What advice would you give other doctors who want to get started, or who are just starting out?

“The key is not to prioritize what’s on your schedule, but to schedule your priorities.”

– Stephen Covey …………… it’s just a really good quote!

Is there anything else you’d like to add that might be interesting to readers?

Most people feel as if change is something that “happens to them”, it is not. One of the most constant themes in life is change. When you notice change starting to take place it is a sign that you need to grow.

Who would you love for us to track down to answer these same questions? 

  • Dr. Marina Del Rios @DraCoquiMD
  • Dr. Andy Little @andyglittle

How I Work Smarter: Molly Estes, MD, FACEP, FAAEM

One word that best describes how you work?

Efficiently

Current mobile device

iPhone 11 Pro

Computer

MacBook Air

What is something you are working on now?

This survey :) In seriousness, trying to complete a survey-based study on the effects of virtual rotations on the 2020-21 EM application cycle.

How did you come up with this Idea/Project?

This idea was generated by a subcommittee I am a part of through the Advising Students Committee in EM (ASCEM) through CORD. After brainstorming about the effects that the required conversion to virtual rotations might have had on this year’s students applying into EM, and how residency programs might view or interpret those “clinical experiences,” we decided to try to gather more data to not only help in the development of future virtual rotations but to also help establish a foundation on which virtual rotation experiences might be able to be interpreted.

What’s your office workspace setup like?

Desk with plenty of space for spreading out papers. Elevated second monitor with wireless full keyboard and mouse. Laptop stand for my MacBook to plug into my second monitor.

 

What’s your best time-saving tip in the office or home?

Honestly, upgrading to a second monitor and a full keyboard has made my work-life infinitely better! It allows me to work more efficiently just having more space to have multiple applications or documents open and visible at a time.

What’s your best time-saving tip regarding email management?

Stay on top of it! I try to buy into the “zero-inbox” philosophy. First and foremost, unsubscribe from anything you don’t use/need. Second, any of the “trash” emails you get from your institution, faculty group, etc. that you don’t read or need, immediately delete. Then use separate folders to archive messages you’ve responded to but want to hold onto. I do keep some emails in my inbox that include “to-do” items, but I try to daily or every-other-day clear these out too.

What apps do you use to keep yourself organized?

Nothing fancy, I have a running Google doc that includes to-do lists (for work and daily life), long-running projects with remaining steps (I tend to bold the next immediate step or the absolutely-have-to-do-today action item), or just thoughts and ideas. At the very bottom of my Google doc currently is a list of dim-sum restaurants I want to try :) I basically use this document as a second brain, I offload anything that I don’t want to have to constantly be thinking about.

How do you stay up to date with resources?

For current events, I get a daily email from theSkimm. I also subscribe to Journal Feed to stay on top of highlighted new research and publications. I’m an active member of AAEM, CORD, etc., and routinely go to conferences so I usually don’t have to worry about CME.

What’s your best time-saving tip in the ED?

Two tips. First, use the resources that you have. I lean heavily on my social workers, case managers, ED techs, etc. to help offload tasks (within appropriate means of course, but I don’t have to teach the patient how to use crutches if my tech is available to do it). Second, batch tasks. If you have to walk all the way across the department to check on a patient, make sure you also check on the other two patients along the way. Or instead of following up on each individual lab result, do other tasks for 10-15 minutes then review all your results at the same time. Recruit others to help too. If I need to supervise one of my residents with a procedure, I ask them to get all their supplies at the bedside, and if I’m not there in 15 minutes then to come grab me. It allows you to keep multiple tasks all in parallel processing.

ED charting: Macros or no macros?

Eh…. I’m not a big macro person. I have a standard one I use for general ROS and PE by age, but I don’t have extensive macros for HPI or MDMs.

Advice

  • What’s the best advice you’ve ever received about work, life, or being efficient?

    Some of the best life advice I ever got was at the end of med school: “Not every decision has a right and a wrong answer; sometimes there is just a decision.” I feel like particularly in medicine we are trained to view decisions in a multiple-choice format, a series of “wrong” answers and a single “right” answer. For me personally, this advice was freeing. No longer do I have to agonize over every little thing, I can make a decision accepting that every option has both good and bad built into it, and then move on with my life. I don’t have to spend 10 minutes debating if the appropriate admitting service is Cards or Medicine, I can just choose one and trust that if I happen to be wrong no harm has come to the patient. It wasn’t “wrong” there was just a different option.

  • What advice would you give other doctors who want to get started, or who are just starting out?

It’s worth it! The hard work, the committee you’re not sure you want to be a part of, volunteering for that one thing, saying hello to that one person, it is all worth it. Especially when you are just starting out, don’t be afraid to put yourself out there. Now granted, try hard not to over-commit, that way lies burnout. But you would be surprised by the number of connections you make, the opportunities that come your way, just simply by your being around and available.

  • Is there anything else you’d like to add that might be interesting to readers?

Find yourself, mentors. And I specifically say mentors plural because you need more than one. It’s fine to have a mentor for your personal life, a mentor for your work life, and a mentor for your academic life. But make sure you have people around you to support you and help you keep growing.

Who would you love for us to track down to answer these same questions?

  • Tim Young, MD @timpyoung
  • Liz Fierro, MD @liz_fierro_md

Read other How I Work Smarter posts, sharing efficiency tips and life advice.

Go to Top