Article review: Inaccuracy in the SLOR

Residency interview season is quickly approaching!

Unique to the field of EM, letters of recommendations from EM faculty are written on a standardized form. The Standardized Letter of Recommendation (SLOR), downloadable from the CORD website, documents information about the student’s performance in the EM clerkship, qualifications, and global assessment. At the end, the letter writer can provide free-text written comments.

In the Global Assessment section, one question is:

How highly would you estimate the candidate will reside on your match list?

The tiered choices include:

“Very Competitive“: If you anticipate the student being in the 2X position of your program’s Rank Order List, where X is the number of PGY-1 positions available in your program
“Competitive“: If you anticipate the student being 2X-4X
“Possible match“: If you anticipate the student being 4X-6X
“Unlikely to match“: If you anticipate the student being greater than 6X

At a previous CORD Academic Assembly, I was lamenting with a few of my EM friends like Dr. Leslie Oyama (UCSD) about how inconsistently letter writers follow these guidelines. Overall, we felt that faculty inflated the student’s Global Assessment Score (GAS) tier.

So we set out to figure how whether GAS tiers were truly correlated with the Rank Order List. Our results were just published in the CORD/CDEM educational supplement of the Academic Emergency Medicine journal.

Methodology

Our study was a multicenter retrospective study of 5 residency programs during the 2008-09 residency application cycle. For each SLOR written by the program’s faculty, the GAS tier was recorded and compared to that student’s actual Rank Order List position.

Results

Of the 102 SLORs, only 26% (n=27) of the SLORs documented a GAS tier that accurately predicted the student’s actual position on the Rank Order List. The student’s position was overestimated in 66% (n=67) of the SLORs and, interestingly, underestimated in 8% (n=8) of the SLORs.

Why the inaccuracies?

Our author group felt that there were 3 primary reasons for these inaccuracies between the GAS tier and Rank Order List

It’s not obvious on the SLOR form that Global Assessment Scores are supposed to be based on the student’s anticipated Rank Order List position. Novice letter writers may not know of the grading scheme. It’s only described on the CORD website.
Determination of a student’s position on the Rank Order List is a multifactorial process. Factors include the EM clerkship grades, USMLE scores, other letters of recommendations, preclinical and clinical grades, and extracurricular activities. One of the most important factors is the Dean’s Letter (Medical Student Performance Evaluation). Most letter writers don’t have access to this document when writing the SLOR. The Dean’s Letter would have comments about disciplinary actions, failed exams, and other “red flag” instances.
Faculty want to advocate for their student in the residency match process. They may feel pressured to inflate the GAS tier. For instance, while the student may be in the 6X+ range on the Rank Order List, some faculty may feel that labeling them in the lowest category of “Unlikely to Match” would unfairly penalize the application of a solid applicant.

What now?

Personally, I think that the Global Assessment Score of the SLOR document is unnecessary. Because of its inaccuracies, it’s hard for the letter reader to put much weight in it.

Additionally, there’s already another assessment tool in the SLOR which essentially gives the letter reader a sense of the student’s overall competitiveness (see below). It’s especially helpful because you can determine whether the letter writer is a “grade-inflater”, because the letter writer has to break down how many letters s/he has written last year within the categories of Outstanding, Excellent, Very Good, and Good. Grade-inflaters, for instance, may have a track record where all of their students fall into the Outstanding category.

References
Oyama L, Kwon M, Fernandez J, Fernández-Frackelton M, Campagne D, Castillo E, Lin M. Inaccuracy of the Global Assessment Score in the Emergency Medicine Standard Letter of Recommendation Acad Emerg Med, 2010; 17:S38-S41. DOI: 10.1111/j.1553-2712.2010.00882.x–