Instructional MethodsNCT04130607
A Study to Evaluate Strategies for Teaching Effective Use of Diagnostic Tests
A study testing Conceptual teaching in people with Instructional Methods, run by Sentara Norfolk General Hospital at 1 site in United States.
Trial summary
A recent Institute of Medicine monograph brought attention to high rates of diagnostic error and called for better educational efforts to improve diagnostic accuracy.1 Educational methods, however, are rarely tested and some educational efforts may be ineffective and wasteful.2 In this study, we plan to examine… Read more about this study
- Who is it for?
- Ages 18 years and olderHealthy volunteers can take part
- Conditions the study is for
- Instructional Methods
- What is being tested?
- Conceptual teaching
- Could I receive a placebo?
- Yes, some participants may receive a placebo
- How many people
- 65 participants
- Where
- 1 siteUnited States
- Study dates
- 15 May 2018to 15 Oct 2019
- Phase
- Interventional
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages 18 years and older
- All sexes
- Healthy volunteers accepted
- Adult, Older adult
Who may be able to join
- Medical Student at McMaster University or Eastern Virginia Medical School
- Completed 18 months of coursework
Who cannot join
None listed.
Trial locations and contacts
1 site in 1 country.
United States1 site
Sentara Norfolk General Hospital
Norfolk, Virginia, 23507
About this study
A recent Institute of Medicine monograph brought attention to high rates of diagnostic error and called for better educational efforts to improve diagnostic accuracy.1 Educational methods, however, are rarely tested and some educational efforts may be ineffective and wasteful.2 In this study, we plan to examine whether explicit instruction on diagnostic methods will have an effect on diagnostic accuracy of 2nd-year medical students and internal medicine residents.
Read the detailed description
Research has shown that expert diagnosticians use a two-step process to confirm a diagnosis: hypothesis generation to generate diagnostic possibilities, followed by hypothesis verification to confirm the most likely diagnostic possibility.3-5 The first step appears to be non-analytical, related to pattern recognition. The second step could be calculated using analytical reasoning, however, physicians rarely make an overt calculation of conditional probabilities. Instead, experienced clinicians typically use an implicit habit or heuristic called "anchoring and adjusting" to incorporate diagnostic testing information into their thinking.6,7 Cognitive psychologists have postulated that anchoring and adjusting provides a way that probability estimates can be updated based on additional new evidence. Most of the discussion in the literature focuses on how this heuristic can lead to biased thinking because of base-rate neglect or anchoring.6 Very little discussion is on how this heuristic could be improved to yield more accurate probability estimates and whether proper use of the heuristic could be taught.
The degree to which a diagnostic test should lead to an adjustment of a probability estimate depends on the operating characteristics of a test, that is, the sensitivity and specificity. Likelihood ratios, once understood, are easier to incorporate into one's thinking, and thus could be used to calibrate the anchoring and adjusting heuristic.7
In this randomized trial, we tested whether explicit conceptual instruction on Bayesian reasoning and likelihood ratios would improve Bayesian updating, compared with a second intervention where we provided multiple (27) examples of clinical problem solving. The third arm provided minimal teaching about diagnosis, but no explicit teaching or examples.
- Conditions
- Instructional Methods
- Being tested
- Conceptual teaching
- Keywords
- Teaching Strategies, Diagnostic reasoning
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
Conceptual teachingOther
Also called Teaching through examples, No active teaching
The present study is designed to contrast two instructional methods - explicit instruction in likelihood ratios and pretest/posttest probabilities versus implicit instruction based on presentation of multiple cases. These will be compared to a "no intervention" control group.
| Study group | Role | What they receive |
|---|---|---|
| Analytical | Experimental | Students will receive brief instruction in probability, sensitivity, specificity, and likelihood ratios, with distributions and calculations. Pretest and posttest probabilities will be computed for two cases for each of the three conditions listed above. Receives: Other: Conceptual teaching |
| Experiential | Active comparator | Students will receive a brief instruction conceptually discussing sensitivity and specificity (e.g. "a sensitive test will be positive at even low levels of disease. However, this can lead to a number of false positive errors, when the test is positive even when there is no disease. As a result, it is most useful for ruling out a diagnosis"). They will then work through a total of 30 cases, 10 for each condition, in blocked sequence. For each brief written case they will be asked for a probability of diagnosis after the clinical information is presented. The test result will then be given and they will be asked for a post-test probability. Their estimate will be compared to the computed value based on published estimates of sensitivity and specificity and feedback provided. Receives: Other: Conceptual teaching |
| No Explicit Instruction or Examples | Placebo comparator | Students will receive 3 passages from a clinical text related to each of the 3 conditions in the study and asked to study them for 15 min each. Receives: Other: Conceptual teaching |
What is being measured
The primary outcome is the main question the study is designed to answer.
Accuracy of participants probability revisions were compared to posttest probability revisions that were calculated using Bayes Rule. An effect size was calculated to measure how close students matched the calculated revision.
To perform the effect size analysis, two transformations were performed. First, the difference between the subjective estimate and the Bayesian calculation of post-test probability was squared to remove negative differences and permit combining of the effects of positive and negative test results. Second, a correction based on the intrinsic error of a probability estimate was applied by dividing each squared difference by p(1-p). In this manner, we transformed each raw difference to a squared effect size (difference / error of difference). Finally, the square root was computed, to transform the data back to an effect size. The resulting effect size was then used for statistical analysis. For this primary analysis, a mixed model ANOVA was used.
Measured: Post-test was taken within 72 hours of instructional phase completion.
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted17 Oct 2019
- Study start15 May 2018
- Primary completion1 Jan 2019
- Study completion15 Oct 2019
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Publications (1)Papers the sponsor has linked to this study.
- Other trials for Instructional MethodsStudies of the same condition, open and under way.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Health services research
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Model description
- Randomized trial of teaching methods with 3 arms.
- Masking
- Single blind
- Who is masked
- Participant
- Enrolment
- 65 participants (actual)
Sponsor and investigators
- Lead sponsor
- Sentara Norfolk General Hospital
- Sponsor type
- University, hospital or other
- Collaborators
- McMaster University
- Responsible party
- Principal investigator, John Brush, Sentara Norfolk General Hospital
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT04130607
- Sponsor's protocol ID
- 18-04-EX-0062
- Status last verified
- Oct 2019
- First submitted
- 15 Oct 2019
- Last updated
- 17 Oct 2019
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- No
Publications
Brush JE Jr, Lee M, Sherbino J, Taylor-Fishwick JC, Norman G. Effect of Teaching Bayesian Methods Using Learning by Concept vs Learning by Example on Medical Students' Ability to Estimate Probability of a Diagnosis: A Randomized Clinical Trial. JAMA Netw Open. 2019 Dec 2;2(12):e1918023. doi: 10.1001/jamanetworkopen.2019.18023.
PubMed 31860107
Study at a glance
From the registry record, last updated by the sponsor on 17 Oct 2019. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- A Study to Evaluate Strategies for Teaching Effective Use of Diagnostic Tests
- ClinicalTrials.gov ID
- NCT04130607
- Acronym / sponsor ID
- 18-04-EX-0062
- Conditions
- Instructional Methods
- Keywords
- Teaching Strategies, Diagnostic reasoning
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 65 participants (actual)
- Lead sponsor
- Sentara Norfolk General HospitalUniversity, hospital or other
- Collaborators
- McMaster University
- Study start
- 15 May 2018 (actual)
- Primary completion
- 1 Jan 2019 (actual)
- Study completion
- 15 Oct 2019 (actual)
- First posted
- 17 Oct 2019 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
A recent Institute of Medicine monograph brought attention to high rates of diagnostic error and called for better educational efforts to improve diagnostic accuracy.1 Educational methods, however, are rarely tested and some educational efforts may be ineffective and wasteful.2 In this study, we plan to examine whether explicit instruction on diagnostic methods will have an effect on diagnostic accuracy of 2nd-year medical students and internal medicine residents.
Detailed description
Research has shown that expert diagnosticians use a two-step process to confirm a diagnosis: hypothesis generation to generate diagnostic possibilities, followed by hypothesis verification to confirm the most likely diagnostic possibility.3-5 The first step appears to be non-analytical, related to pattern recognition. The second step could be calculated using analytical reasoning, however, physicians rarely make an overt calculation of conditional probabilities. Instead, experienced clinicians typically use an implicit habit or heuristic called "anchoring and adjusting" to incorporate diagnostic testing information into their thinking.6,7 Cognitive psychologists have postulated that anchoring and adjusting provides a way that probability estimates can be updated based on additional new evidence. Most of the discussion in the literature focuses on how this heuristic can lead to biased thinking because of base-rate neglect or anchoring.6 Very little discussion is on how this heuristic could be improved to yield more accurate probability estimates and whether proper use of the heuristic could be taught.
The degree to which a diagnostic test should lead to an adjustment of a probability estimate depends on the operating characteristics of a test, that is, the sensitivity and specificity. Likelihood ratios, once understood, are easier to incorporate into one's thinking, and thus could be used to calibrate the anchoring and adjusting heuristic.7
In this randomized trial, we tested whether explicit conceptual instruction on Bayesian reasoning and likelihood ratios would improve Bayesian updating, compared with a second intervention where we provided multiple (27) examples of clinical problem solving. The third arm provided minimal teaching about diagnosis, but no explicit teaching or examples.
Study design
- Primary purpose
- Health services research
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Masking
- Single blind (participant)
- Number of arms
- 3
- Model description
- Randomized trial of teaching methods with 3 arms.
| Arm | Type | Receives |
|---|---|---|
| Analytical | Experimental | Other: Conceptual teachingStudents will receive brief instruction in probability, sensitivity, specificity, and likelihood ratios, with distributions and calculations. Pretest and posttest probabilities will be computed for two cases for each of the three conditions listed above. |
| Experiential | Active comparator | Other: Conceptual teachingStudents will receive a brief instruction conceptually discussing sensitivity and specificity (e.g. "a sensitive test will be positive at even low levels of disease. However, this can lead to a number of false positive errors, when the test is positive even when there is no disease. As a result, it is most useful for ruling out a diagnosis"). They will then work through a total of 30 cases, 10 for each condition, in blocked sequence. For each brief written case they will be asked for a probability of diagnosis after the clinical information is presented. The test result will then be given and they will be asked for a post-test probability. Their estimate will be compared to the computed value based on published estimates of sensitivity and specificity and feedback provided. |
| No Explicit Instruction or Examples | Placebo comparator | Other: Conceptual teachingStudents will receive 3 passages from a clinical text related to each of the 3 conditions in the study and asked to study them for 15 min each. |
Who may qualify
- Ages
- 18 years and older
- Sex
- All
- Healthy volunteers
- Accepted
- Age groups
- Adult, Older adult
Key inclusion criteria (2 listed)
- Medical Student at McMaster University or Eastern Virginia Medical School
- Completed 18 months of coursework
Key exclusion criteria (0 listed)
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| Conceptual teachingAlso: Teaching through examples, No active teachingThe present study is designed to contrast two instructional methods - explicit instruction in likelihood ratios and pretest/posttest probabilities versus implicit instruction based on presentation of multiple cases. These will be compared to a "no intervention" control group. | Other | Analytical, Experiential, No Explicit Instruction or Examples |
Primary outcome
Accuracy of participants probability revisions were compared to posttest probability revisions that were calculated using Bayes Rule. An effect size was calculated to measure how close students matched the calculated revision.
Time frame: Post-test was taken within 72 hours of instructional phase completion.
To perform the effect size analysis, two transformations were performed. First, the difference between the subjective estimate and the Bayesian calculation of post-test probability was squared to remove negative differences and permit combining of the effects of positive and negative test results. Second, a correction based on the intrinsic error of a probability estimate was applied by dividing each squared difference by p(1-p). In this manner, we transformed each raw difference to a squared effect size (difference / error of difference). Finally, the square root was computed, to transform the data back to an effect size. The resulting effect size was then used for statistical analysis. For this primary analysis, a mixed model ANOVA was used.
Results and safety
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 17 Oct 2019.
- Derived from this study: PubMed 31860107Brush JE Jr, Lee M, Sherbino J, Taylor-Fishwick JC, Norman G. Effect of Teaching Bayesian Methods Using Learning by Concept vs Learning by Example on Medical Students' Ability to Estimate Probability of a Diagnosis: A Randomized Clinical Trial. JAMA Netw Open. 2019 Dec 2;2(12):e1918023. doi: 10.1001/jamanetworkopen.2019.18023.
- Individual participant dataSharing planned: No
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
The record lists no contact. Use the official record to reach the sponsor.
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing Conceptual teaching in people with Instructional Methods, run by Sentara Norfolk General Hospital at 1 site in United States.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 2 inclusion and 0 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 17 Oct 2019. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in United States. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. Some participants in this study may receive a placebo. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Accuracy of participants probability revisions were compared to posttest probability revisions that were calculated using Bayes Rule. An effect size was calculated to measure how close students matched the calculated revision. (time frame: Post-test was taken within 72 hours of instructional phase completion.). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT04130607, which also lists 1 linked publication.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 17 Oct 2019.