Internship and ResidencyBRIARNCT05616065
Bias Reduction in Academic Recruitment
A study testing Blinding to academic performance of candidate and Usual Interview methodology in people with Internship and Residency and Interview, run by Community Medical Center, Toms River, NJ at 1 site in United States.
Trial summary
The aim of this study is to determine if, with respect to medical students applying for residency in emergency medicine, prior knowledge of an applicant's academic record affects their ranking during their interviews. Read more about this study
- Who is it for?
- Any ageHealthy volunteers can take part
- Conditions the study is for
- Internship and Residency, Interview, Education, Medical
- What is being tested?
- Blinding to academic performance of candidate and Usual Interview methodology
- Could I receive a placebo?
- No, everyone receives an active treatment
- How many people
- 212 participants
- Where
- 1 siteUnited States
- Study dates
- 12 Oct 2022to 8 Feb 2023
- Phase
- Interventional
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages Any age
- All sexes
- Healthy volunteers accepted
- Child, Adult, Older adult
Who may be able to join
- All candidates who have or are anticipating completing their medical school education, and have applied for a position in our residency during the 2022-2023 match cycle.
Who cannot join
- Candidates our program declined to grant an interview spot or have canceled their interview prior to completing it.
Trial locations and contacts
1 site in 1 country.
United States1 site
Community Medical Center
Toms River, New Jersey, 08755
About this study
The aim of this study is to determine if, with respect to medical students applying for residency in emergency medicine, prior knowledge of an applicant's academic record affects their ranking during their interviews.
Attendings interviewing candidates will either be blinded to their prior academic records or be allowed to review them prior to the interview. The interviewer's final score of the applicant will be measured, comparing blinded versus unblinded assessors.
Read the detailed description
In residency recruitment and ranking, the most honest admission educators can offer is that there truly is no exact way to identify candidates who will thrive in a given discipline or residency program. Along with applications and letters of reference, the traditional residency interview process has long been thought to be one of the best available tools to evaluate a given candidate as well as assess their 'fit' for the culture of the program they wish to enter. Despite this decades-used tool, little uniformity exists in practice, with various disciplines and programs using various techniques to evaluate potential residents. To wit, prospective resident themselves, in one study, have listed their top three consideration when selecting a residency to be the friendliness of the program and faculty, the overall environment, and the interview day itself. Spending time in the department and hospital and having contact with the faculty and staff has long been seen as the best way to 'get to know' the program. This is among the main reasons why medical students will often schedule 'audition rotations' at hospitals and programs that draw their interest.
With respect to the interviews themselves, there has been some evolution of thought, despite the variability. Recently, structured interviews and other techniques have been discussed to reduce bias and increase efficiency in residency recruitment.
While this has been discussed in academic circles for some time, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has brought these issues to the fore. Forcing interviews to be done remotely has effectively removed the intimacy of an in-person visit or interview, making an assessment of 'fit' for both the applicant and the program more challenging. Many program leaders and resident educators have described 'fit' as among the most important predictors of residency success, sometimes more so than pure academic prowess.
Of the many biases in residency interviewing, among the easiest and most seductive is hinging a decision on academic achievement alone. It is well understood by many residency educators that academic success in medical school does not always translate to the same level of performance in residency. With the virtual method of interviews continuing for at least this year (and likely going forward), our study ponders the question if the removal of knowledge of an applicant's academic record changes their interviewer's perception of their potential as a resident in an emergency medicine.
This study will examine if blinding interviewers to the academic performance of residency applicants impact the assessor's recommendation for the candidate's match day rank list placement. Candidates will be assigned to have one of three of their interviews performed during the standard interview process, to have a single assessor who is instructed not to look at either medical school transcripts or standardized test scores. Assessors will be permitted to look at letters of recommendation, personal statements, research and work experience, and hobbies.
The absolute value of the difference of between blinded and unblinded assessors will be compared and tested for statistical significance by matched student T testing. The study is powered for an absolute difference of 1 point on a ten-point scale, which translates linearly to rank list placement. Additionally, this 1-point difference is considered significant from a standpoint of residency operations. Also, blinded versus unblinded assessors will be compared to the placement by the residency program director, who will remain unblinded and also makes independent recommendations for every candidate. As a subanalysis, a qualitative assessment of candidates who get ranked either substantially higher or lower when blinded compared to unblinded will also be conducted.
- Conditions
- Internship and Residency, Interview, Education, Medical
- Being tested
- Blinding to academic performance of candidate, Usual Interview methodology
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
Blinding to academic performance of candidateOther
Each candidate applying to our program will be interviewed by three core faculty. One of them will be selected, a priori, to be asked not to view a candidate's academic performance (medical school transcripts, standardized test scores). This will not be randomized. Instead, it shall be set up so that each assessor has two blinded candidates to interview each day scheduled. Given that different faculty will be scheduled each day, it may end up that certain faculty will be blinded more than others.
Usual Interview methodologyOther
Each candidate applying to our program will be interviewed by three core faculty. Two of them will be selected, a priori, to be asked to interview their candidates as they see fit.
| Study group | Role | What they receive |
|---|---|---|
| Blinded | Experimental | Interviewers will be assigned a candidate to interview. They will be asked not to review the medical school transcripts or standardized test scores (USMLE, COMLEX). They will be permitted to review personal statements, letters or recommendation, and other personal information in the application. Receives: Other: Blinding to academic performance of candidate |
| Unblinded | Active comparator | Interviewers will be assigned a candidate to interview. They will be given full permission to review the full application of their candidate as per their normal interview protocol. Receives: Other: Usual Interview methodology |
What is being measured
The primary outcome is the main question the study is designed to answer.
Assessor recommendation for match list position
Assessors will be asked to score their applicant, 1-10, with 1 corresponding to a Do Not Rank recommendation, 2-4 with lower third of list, 5-7 middle third of list, and 8-10 upper third of list.
Measured: Immediately after interview
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted15 Nov 2022
- Study start12 Oct 2022
- Primary completion8 Feb 2023
- Study completion8 Feb 2023
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Publications (12)Papers the sponsor has linked to this study.
- Other trials for Internship and ResidencyStudies of the same condition, open and under way.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Health services research
- Allocation
- Non-randomised
- Intervention model
- Parallel groups
- Masking
- None (open label)
- Enrolment
- 212 participants (actual)
Sponsor and investigators
- Lead sponsor
- Community Medical Center, Toms River, NJ
- Sponsor type
- University, hospital or other
- Responsible party
- Sponsor
- Investigators
- Joel Kravitz, MD, Principal investigator, Associate Program Director, Emergency Medicine Residency
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT05616065
- Sponsor's protocol ID
- 22-029
- Acronym
- BRIAR
- Status last verified
- Apr 2023
- First submitted
- 7 Nov 2022
- Last updated
- 19 Apr 2023
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- No
Publications
Kiraly L, Dewey E, Brasel K. Hawks and Doves: Adjusting for Bias in Residency Interview Scoring. J Surg Educ. 2020 Nov-Dec;77(6):e132-e137. doi: 10.1016/j.jsurg.2020.08.013. Epub 2020 Aug 27.
PubMed 32863174Poon S, Nellans K, Rothman A, Crabb RAL, Wendolowski SF, Kiridly D, Gecelter R, Gorroochurn P, Chahine NO. Underrepresented Minority Applicants Are Competitive for Orthopaedic Surgery Residency Programs, but Enter Residency at Lower Rates. J Am Acad Orthop Surg. 2019 Nov 1;27(21):e957-e968. doi: 10.5435/JAAOS-D-17-00811.
PubMed 30614894Hughes RH, Kleinschmidt S, Sheng AY. Using structured interviews to reduce bias in emergency medicine residency recruitment: Worth a second look. AEM Educ Train. 2021 Sep 1;5(Suppl 1):S130-S134. doi: 10.1002/aet2.10562. eCollection 2021 Sep. No abstract available.
PubMed 34616987Williams AL, Blomkalns AL, Gibler WB. Residency training in emergency medicine: the challenges of the 21st century. Keio J Med. 2004 Dec;53(4):203-9. doi: 10.2302/kjm.53.203.
PubMed 15647626Balhara KS, Weygandt PL, Ehmann MR, Regan L. Navigating Bias on Interview Day: Strategies for Charting an Inclusive and Equitable Course. J Grad Med Educ. 2021 Aug;13(4):466-470. doi: 10.4300/JGME-D-21-00001.1. Epub 2021 Aug 13. No abstract available.
PubMed 34434507Theiss LM, Prather JC, Porterfield JR, Corey B, Chen H, McGwin G, Johnson MD, Theiss SM. Prevalence, Bias, and Rank List Impact of Illegal Questions in Surgical Specialty Residency Interviews. J Surg Educ. 2022 Jan-Feb;79(1):69-76. doi: 10.1016/j.jsurg.2021.07.015. Epub 2021 Aug 13.
PubMed 34400121DeSantis M, Marco CA. Emergency medicine residency selection: factors influencing candidate decisions. Acad Emerg Med. 2005 Jun;12(6):559-61. doi: 10.1197/j.aem.2005.01.006.
PubMed 15930408Smilen SW, Funai EF, Bianco AT. Residency selection: should interviewers be given applicants' board scores? Am J Obstet Gynecol. 2001 Feb;184(3):508-13. doi: 10.1067/mob.2001.109868.
PubMed 11228511Haas MRC, He S, Sternberg K, Jordan J, Deiorio NM, Chan TM, Yarris LM. Reimagining Residency Selection: Part 1-A Practical Guide to Recruitment in the Post-COVID-19 Era. J Grad Med Educ. 2020 Oct;12(5):539-544. doi: 10.4300/JGME-D-20-00907.1. No abstract available.
PubMed 33149819Sternberg K, Jordan J, Haas MRC, He S, Deiorio NM, Yarris LM, Chan TM. Reimagining Residency Selection: Part 2-A Practical Guide to Interviewing in the Post-COVID-19 Era. J Grad Med Educ. 2020 Oct;12(5):545-549. doi: 10.4300/JGME-D-20-00911.1. No abstract available.
PubMed 33149820Jordan J, Sternberg K, Haas MRC, He S, Yarris LM, Chan TM, Deiorio NM. Reimagining Residency Selection: Part 3-A Practical Guide to Ranking Applicants in the Post-COVID-19 Era. J Grad Med Educ. 2020 Dec;12(6):666-670. doi: 10.4300/JGME-D-20-01087.1. Epub 2020 Dec 18. No abstract available.
PubMed 33391587Stephenson-Famy A, Houmard BS, Oberoi S, Manyak A, Chiang S, Kim S. Use of the Interview in Resident Candidate Selection: A Review of the Literature. J Grad Med Educ. 2015 Dec;7(4):539-48. doi: 10.4300/JGME-D-14-00236.1.
PubMed 26692964
Study at a glance
From the registry record, last updated by the sponsor on 19 Apr 2023. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Bias Reduction in Academic Recruitment
- ClinicalTrials.gov ID
- NCT05616065
- Acronym / sponsor ID
- BRIAR / 22-029
- Conditions
- Internship and Residency, Interview, Education, Medical
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 212 participants (actual)
- Lead sponsor
- Community Medical Center, Toms River, NJUniversity, hospital or other
- Study start
- 12 Oct 2022 (actual)
- Primary completion
- 8 Feb 2023 (actual)
- Study completion
- 8 Feb 2023 (actual)
- First posted
- 15 Nov 2022 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
The aim of this study is to determine if, with respect to medical students applying for residency in emergency medicine, prior knowledge of an applicant's academic record affects their ranking during their interviews.
Attendings interviewing candidates will either be blinded to their prior academic records or be allowed to review them prior to the interview. The interviewer's final score of the applicant will be measured, comparing blinded versus unblinded assessors.
Detailed description
In residency recruitment and ranking, the most honest admission educators can offer is that there truly is no exact way to identify candidates who will thrive in a given discipline or residency program. Along with applications and letters of reference, the traditional residency interview process has long been thought to be one of the best available tools to evaluate a given candidate as well as assess their 'fit' for the culture of the program they wish to enter. Despite this decades-used tool, little uniformity exists in practice, with various disciplines and programs using various techniques to evaluate potential residents. To wit, prospective resident themselves, in one study, have listed their top three consideration when selecting a residency to be the friendliness of the program and faculty, the overall environment, and the interview day itself. Spending time in the department and hospital and having contact with the faculty and staff has long been seen as the best way to 'get to know' the program. This is among the main reasons why medical students will often schedule 'audition rotations' at hospitals and programs that draw their interest.
With respect to the interviews themselves, there has been some evolution of thought, despite the variability. Recently, structured interviews and other techniques have been discussed to reduce bias and increase efficiency in residency recruitment.
While this has been discussed in academic circles for some time, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has brought these issues to the fore. Forcing interviews to be done remotely has effectively removed the intimacy of an in-person visit or interview, making an assessment of 'fit' for both the applicant and the program more challenging. Many program leaders and resident educators have described 'fit' as among the most important predictors of residency success, sometimes more so than pure academic prowess.
Of the many biases in residency interviewing, among the easiest and most seductive is hinging a decision on academic achievement alone. It is well understood by many residency educators that academic success in medical school does not always translate to the same level of performance in residency. With the virtual method of interviews continuing for at least this year (and likely going forward), our study ponders the question if the removal of knowledge of an applicant's academic record changes their interviewer's perception of their potential as a resident in an emergency medicine.
This study will examine if blinding interviewers to the academic performance of residency applicants impact the assessor's recommendation for the candidate's match day rank list placement. Candidates will be assigned to have one of three of their interviews performed during the standard interview process, to have a single assessor who is instructed not to look at either medical school transcripts or standardized test scores. Assessors will be permitted to look at letters of recommendation, personal statements, research and work experience, and hobbies.
The absolute value of the difference of between blinded and unblinded assessors will be compared and tested for statistical significance by matched student T testing. The study is powered for an absolute difference of 1 point on a ten-point scale, which translates linearly to rank list placement. Additionally, this 1-point difference is considered significant from a standpoint of residency operations. Also, blinded versus unblinded assessors will be compared to the placement by the residency program director, who will remain unblinded and also makes independent recommendations for every candidate. As a subanalysis, a qualitative assessment of candidates who get ranked either substantially higher or lower when blinded compared to unblinded will also be conducted.
Study design
- Primary purpose
- Health services research
- Allocation
- Non-randomised
- Intervention model
- Parallel groups
- Masking
- None (open label)
- Number of arms
- 2
| Arm | Type | Receives |
|---|---|---|
| Blinded | Experimental | Other: Blinding to academic performance of candidateInterviewers will be assigned a candidate to interview. They will be asked not to review the medical school transcripts or standardized test scores (USMLE, COMLEX). They will be permitted to review personal statements, letters or recommendation, and other personal information in the application. |
| Unblinded | Active comparator | Other: Usual Interview methodologyInterviewers will be assigned a candidate to interview. They will be given full permission to review the full application of their candidate as per their normal interview protocol. |
Who may qualify
- Sex
- All
- Healthy volunteers
- Accepted
- Age groups
- Child, Adult, Older adult
Key inclusion criteria (1 listed)
- All candidates who have or are anticipating completing their medical school education, and have applied for a position in our residency during the 2022-2023 match cycle.
Key exclusion criteria (1 listed)
- Candidates our program declined to grant an interview spot or have canceled their interview prior to completing it.
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| Blinding to academic performance of candidateEach candidate applying to our program will be interviewed by three core faculty. One of them will be selected, a priori, to be asked not to view a candidate's academic performance (medical school transcripts, standardized test scores). This will not be randomized. Instead, it shall be set up so that each assessor has two blinded candidates to interview each day scheduled. Given that different faculty will be scheduled each day, it may end up that certain faculty will be blinded more than others. | Other | Blinded |
| Usual Interview methodologyEach candidate applying to our program will be interviewed by three core faculty. Two of them will be selected, a priori, to be asked to interview their candidates as they see fit. | Other | Unblinded |
Primary outcome
Assessor recommendation for match list position
Time frame: Immediately after interview
Assessors will be asked to score their applicant, 1-10, with 1 corresponding to a Do Not Rank recommendation, 2-4 with lower third of list, 5-7 middle third of list, and 8-10 upper third of list.
Results and safety
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 19 Apr 2023.
- Background: PubMed 32863174Kiraly L, Dewey E, Brasel K. Hawks and Doves: Adjusting for Bias in Residency Interview Scoring. J Surg Educ. 2020 Nov-Dec;77(6):e132-e137. doi: 10.1016/j.jsurg.2020.08.013. Epub 2020 Aug 27.
- Background: PubMed 30614894Poon S, Nellans K, Rothman A, Crabb RAL, Wendolowski SF, Kiridly D, Gecelter R, Gorroochurn P, Chahine NO. Underrepresented Minority Applicants Are Competitive for Orthopaedic Surgery Residency Programs, but Enter Residency at Lower Rates. J Am Acad Orthop Surg. 2019 Nov 1;27(21):e957-e968. doi: 10.5435/JAAOS-D-17-00811.
- Background: PubMed 34616987Hughes RH, Kleinschmidt S, Sheng AY. Using structured interviews to reduce bias in emergency medicine residency recruitment: Worth a second look. AEM Educ Train. 2021 Sep 1;5(Suppl 1):S130-S134. doi: 10.1002/aet2.10562. eCollection 2021 Sep. No abstract available.
- Background: PubMed 15647626Williams AL, Blomkalns AL, Gibler WB. Residency training in emergency medicine: the challenges of the 21st century. Keio J Med. 2004 Dec;53(4):203-9. doi: 10.2302/kjm.53.203.
- Background: PubMed 34434507Balhara KS, Weygandt PL, Ehmann MR, Regan L. Navigating Bias on Interview Day: Strategies for Charting an Inclusive and Equitable Course. J Grad Med Educ. 2021 Aug;13(4):466-470. doi: 10.4300/JGME-D-21-00001.1. Epub 2021 Aug 13. No abstract available.
- Background: PubMed 34400121Theiss LM, Prather JC, Porterfield JR, Corey B, Chen H, McGwin G, Johnson MD, Theiss SM. Prevalence, Bias, and Rank List Impact of Illegal Questions in Surgical Specialty Residency Interviews. J Surg Educ. 2022 Jan-Feb;79(1):69-76. doi: 10.1016/j.jsurg.2021.07.015. Epub 2021 Aug 13.
- Background: PubMed 15930408DeSantis M, Marco CA. Emergency medicine residency selection: factors influencing candidate decisions. Acad Emerg Med. 2005 Jun;12(6):559-61. doi: 10.1197/j.aem.2005.01.006.
- Background: PubMed 11228511Smilen SW, Funai EF, Bianco AT. Residency selection: should interviewers be given applicants' board scores? Am J Obstet Gynecol. 2001 Feb;184(3):508-13. doi: 10.1067/mob.2001.109868.
- Background: PubMed 33149819Haas MRC, He S, Sternberg K, Jordan J, Deiorio NM, Chan TM, Yarris LM. Reimagining Residency Selection: Part 1-A Practical Guide to Recruitment in the Post-COVID-19 Era. J Grad Med Educ. 2020 Oct;12(5):539-544. doi: 10.4300/JGME-D-20-00907.1. No abstract available.
- Background: PubMed 33149820Sternberg K, Jordan J, Haas MRC, He S, Deiorio NM, Yarris LM, Chan TM. Reimagining Residency Selection: Part 2-A Practical Guide to Interviewing in the Post-COVID-19 Era. J Grad Med Educ. 2020 Oct;12(5):545-549. doi: 10.4300/JGME-D-20-00911.1. No abstract available.
- Background: PubMed 33391587Jordan J, Sternberg K, Haas MRC, He S, Yarris LM, Chan TM, Deiorio NM. Reimagining Residency Selection: Part 3-A Practical Guide to Ranking Applicants in the Post-COVID-19 Era. J Grad Med Educ. 2020 Dec;12(6):666-670. doi: 10.4300/JGME-D-20-01087.1. Epub 2020 Dec 18. No abstract available.
- Background: PubMed 26692964Stephenson-Famy A, Houmard BS, Oberoi S, Manyak A, Chiang S, Kim S. Use of the Interview in Resident Candidate Selection: A Review of the Literature. J Grad Med Educ. 2015 Dec;7(4):539-48. doi: 10.4300/JGME-D-14-00236.1.
- Individual participant dataSharing planned: No
Related trials for Internship and Residency
Recruiting
Medical Residents Learning Weight Management Counseling Skills
Weight Management Counseling
Enrolling by invitation
Measuring AI Reliance Among Intern Doctors in Palestine
Radiology, Internship and Residency
Recruiting
Reaching Out to Distressed Medical Residents, Fellows and Faculty
Depression, Suicide
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
Joel Kravitz, MD principal investigator
Associate Program Director, Emergency Medicine Residency
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing Blinding to academic performance of candidate and Usual Interview methodology in people with Internship and Residency and Interview, run by Community Medical Center, Toms River, NJ at 1 site in United States.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 1 inclusion and 1 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 19 Apr 2023. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in United States. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Assessor recommendation for match list position (time frame: Immediately after interview). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT05616065, which also lists 12 linked publications.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 19 Apr 2023.