DiagnosisNCT06157944
Physician Reasoning on Diagnostic Cases With Large Language Models
A study testing GPT-4 in people with Diagnosis, run by Stanford University at 1 site in United States.
Trial summary
This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional diagnostic decision support tools on performance on case-based diagnostic reasoning tasks. Read more about this study
- Who is it for?
- Any ageHealthy volunteers can take part
- Conditions the study is for
- Diagnosis
- What is being tested?
- GPT-4
- Could I receive a placebo?
- No, everyone receives an active treatment
- How many people
- 50 participants
- Where
- 1 siteUnited States
- Study dates
- 29 Nov 2023to 30 Dec 2023
- Phase
- Interventional
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages Any age
- All sexes
- Healthy volunteers accepted
- Child, Adult, Older adult
Who may be able to join
- Participants must be licensed physicians and have completed at least post-graduate year 2 (PGY2) of medical training.
- Training in Internal medicine, family medicine, or emergency medicine.
Who cannot join
- Not currently practicing clinically.
Trial locations and contacts
1 site in 1 country.
About this study
Official title: Diagnostic Reasoning With Large Language Model Chat Bots
This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional diagnostic decision support tools on performance on case-based diagnostic reasoning tasks.
Read the detailed description
Artificial intelligence (AI) technologies, specifically advanced large language models like OpenAI's ChatGPT, have the potential to improve medical decision-making. Although ChatGPT-4 was not developed for its use in medical-specific applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultation. However, little is known about how ChatGPT augments the clinical reasoning abilities of clinicians.
Clinical reasoning is a complex process involving pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could potentially help reduce clinician workload and decrease the likelihood of missed diagnoses. However, ChatGPT-4 was not developed for the purpose of clinical reasoning nor has it been validated for this purpose. Further, it may be subject to disinformation, including convincing confabulations that may mislead clinicians. If clinicians misuse this tool, it may not improve diagnostic reasoning and could even cause harm. Therefore, it is important to study how clinicians use large language models to augment clinical reasoning prior to routine incorporation into patient care.
In this study, we will randomize participants to answer diagnostic cases with or without access to ChatGPT-4. The participants will be asked to give three differential diagnoses for each case, with supporting and opposing findings for each diagnosis. Additionally they will be asked to provide their top diagnosis along with next diagnostic steps. Answers will be graded by independent reviewers blinded to treatment assignment.
- Conditions
- Diagnosis
- Being tested
- GPT-4
- Keywords
- Computer-assisted diagnosis, Large language models, clinical reasoning
- Medical subject headings
- Disease
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
GPT-4Other
OpenAI's GPT-4 large language model with chat interface.
| Study group | Role | What they receive |
|---|---|---|
| GPT-4 | Active comparator | Group will be given access to GPT-4. Receives: Other: GPT-4 |
| Usual resources | No intervention | Group will not be given access to GPT-4 but will be encouraged to use any resources they wish besides large language models (UpToDate, Dynamed, google, etc). |
What is being measured
The primary outcome is the main question the study is designed to answer.
Diagnostic reasoning
The primary outcome will be the percent correct (range: 0 to 100) for each case. For each case, participants will be asked for three top diagnoses and findings from the case that support that diagnosis and oppose that diagnosis. Participants will receive 1 point for each plausible diagnosis. Findings supporting the diagnosis and findings opposing the diagnosis will also be graded based on correctness, with 1 point for partially correct and 2 points for completely correct responses. Participants will then be asked to name their top diagnosis, earning one point for a reasonable response and two points for the most correct response. Finally participants will be asked to name up to 3 next steps to further evaluate the patient with one point awarded for a partially correct response and two points for a completely correct response. The primary outcome will be compared on the case-level by the randomized groups.
Measured: During evaluation
Secondary outcomes (1)
Time Spent on Diagnosis
We will compare how much time (in minutes) participants spend per case between the two study arms.
Measured: During evaluation
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted6 Dec 2023
- Study start29 Nov 2023
- Primary completion30 Dec 2023
- Study completion30 Dec 2023
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Publications (1)Papers the sponsor has linked to this study.
- Other trials for DiagnosisStudies of the same condition, open and under way.
- More trials run by Stanford UniversityThe sponsor's other registered studies.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Diagnosis
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Model description
- The trial will be designed as a randomized, two-arm, single-blind parallel group study.
- Masking
- Single blind
- Who is masked
- Outcomes assessor
- Enrolment
- 50 participants (actual)
Sponsor and investigators
- Lead sponsor
- Stanford University
- Sponsor type
- University, hospital or other
- Collaborators
- Beth Israel Deaconess Medical Center, University of Minnesota
- Responsible party
- Principal investigator, Jonathan Chen, Stanford University
- Investigators
- Jonathan H Chen, MD, PhD, Principal investigator, Stanford UniversityAdam Rodman, MD, Principal investigator, Beth Israel Deaconess Medical CenterAndrew Olson, MD, Principal investigator, University of Minnesota
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT06157944
- Sponsor's protocol ID
- 71319
- Status last verified
- Feb 2024
- First submitted
- 27 Nov 2023
- Last updated
- 20 Feb 2024
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- No
Publications
Goh E, Gallo R, Hom J, Strong E, Weng Y, Kerman H, Cool JA, Kanjee Z, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Rodman A, Chen JH. Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial. JAMA Netw Open. 2024 Oct 1;7(10):e2440969. doi: 10.1001/jamanetworkopen.2024.40969.
PubMed 39466245
Study at a glance
From the registry record, last updated by the sponsor on 20 Feb 2024. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Diagnostic Reasoning With Large Language Model Chat Bots
- ClinicalTrials.gov ID
- NCT06157944
- Acronym / sponsor ID
- 71319
- Conditions
- Diagnosis
- Keywords
- Computer-assisted diagnosis, Large language models, clinical reasoning
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 50 participants (actual)
- Lead sponsor
- Stanford UniversityUniversity, hospital or other
- Collaborators
- Beth Israel Deaconess Medical Center, University of Minnesota
- Study start
- 29 Nov 2023 (actual)
- Primary completion
- 30 Dec 2023 (actual)
- Study completion
- 30 Dec 2023 (actual)
- First posted
- 6 Dec 2023 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional diagnostic decision support tools on performance on case-based diagnostic reasoning tasks.
Detailed description
Artificial intelligence (AI) technologies, specifically advanced large language models like OpenAI's ChatGPT, have the potential to improve medical decision-making. Although ChatGPT-4 was not developed for its use in medical-specific applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultation. However, little is known about how ChatGPT augments the clinical reasoning abilities of clinicians.
Clinical reasoning is a complex process involving pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could potentially help reduce clinician workload and decrease the likelihood of missed diagnoses. However, ChatGPT-4 was not developed for the purpose of clinical reasoning nor has it been validated for this purpose. Further, it may be subject to disinformation, including convincing confabulations that may mislead clinicians. If clinicians misuse this tool, it may not improve diagnostic reasoning and could even cause harm. Therefore, it is important to study how clinicians use large language models to augment clinical reasoning prior to routine incorporation into patient care.
In this study, we will randomize participants to answer diagnostic cases with or without access to ChatGPT-4. The participants will be asked to give three differential diagnoses for each case, with supporting and opposing findings for each diagnosis. Additionally they will be asked to provide their top diagnosis along with next diagnostic steps. Answers will be graded by independent reviewers blinded to treatment assignment.
Study design
- Primary purpose
- Diagnosis
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Masking
- Single blind (outcomes assessor)
- Number of arms
- 2
- Model description
- The trial will be designed as a randomized, two-arm, single-blind parallel group study.
| Arm | Type | Receives |
|---|---|---|
| GPT-4 | Active comparator | Other: GPT-4Group will be given access to GPT-4. |
| Usual resources | No intervention | –Group will not be given access to GPT-4 but will be encouraged to use any resources they wish besides large language models (UpToDate, Dynamed, google, etc). |
Who may qualify
- Sex
- All
- Healthy volunteers
- Accepted
- Age groups
- Child, Adult, Older adult
Key inclusion criteria (2 listed)
- Participants must be licensed physicians and have completed at least post-graduate year 2 (PGY2) of medical training.
- Training in Internal medicine, family medicine, or emergency medicine.
Key exclusion criteria (1 listed)
- Not currently practicing clinically.
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| GPT-4OpenAI's GPT-4 large language model with chat interface. | Other | GPT-4 |
Primary outcome
Diagnostic reasoning
Time frame: During evaluation
The primary outcome will be the percent correct (range: 0 to 100) for each case. For each case, participants will be asked for three top diagnoses and findings from the case that support that diagnosis and oppose that diagnosis. Participants will receive 1 point for each plausible diagnosis. Findings supporting the diagnosis and findings opposing the diagnosis will also be graded based on correctness, with 1 point for partially correct and 2 points for completely correct responses. Participants will then be asked to name their top diagnosis, earning one point for a reasonable response and two points for the most correct response. Finally participants will be asked to name up to 3 next steps to further evaluate the patient with one point awarded for a partially correct response and two points for a completely correct response. The primary outcome will be compared on the case-level by the randomized groups.
1 secondary outcome
Time Spent on Diagnosis
Time frame: During evaluation
Results and safety
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 20 Feb 2024.
- Derived from this study: PubMed 39466245Goh E, Gallo R, Hom J, Strong E, Weng Y, Kerman H, Cool JA, Kanjee Z, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Rodman A, Chen JH. Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial. JAMA Netw Open. 2024 Oct 1;7(10):e2440969. doi: 10.1001/jamanetworkopen.2024.40969.
- Individual participant dataSharing planned: No
Related trials for Diagnosis
Not yet recruitingPhase 1/2
177Lu-DOTA-EB-TATE in Adult Patients With Metastatic, Radioactive Iodine Non-Responsive Oncocytic (Hurthle-Cell) Thyroid Cancer
Thyroid Cancer
Recruiting
Tissue Modeling in Systemic Sclerosis Using Induced Pluripotent Stem Cells (iPSCs)
Systemic Sclerosis (SSc)
Not yet recruiting
GOT VOICE Feasibility With In-Person Group Singing
Elderly, Coronary Arterial Disease (CAD)
RecruitingObservational study
Investigating Complex Neurodegenerative Disorders Related to Amyotrophic Lateral Sclerosis and Frontotemporal Dementia
Frontotemporal Dementia, Amyotrophic Lateral Sclerosis
RecruitingObservational study
A Longitudinal Study of Familial Hypereosinophilia (FE): Natural History and Markers of Disease Progression
Eosinophilia, Hypereosinophilic Syndrome
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
Jonathan H Chen, MD, PhD principal investigator
Stanford University
Adam Rodman, MD principal investigator
Beth Israel Deaconess Medical Center
Andrew Olson, MD principal investigator
University of Minnesota
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing GPT-4 in people with Diagnosis, run by Stanford University at 1 site in United States.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 2 inclusion and 1 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 20 Feb 2024. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in United States. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Diagnostic reasoning (time frame: During evaluation). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT06157944, which also lists 1 linked publication.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 20 Feb 2024.