Clinical Decision-makingNCT06208423
Physician Reasoning on Management Cases With Large Language Models
A study testing GPT-4 in people with Clinical Decision-making, run by Stanford University at 1 site in United States.
Trial summary
This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional management decision support tools on performance on case-based management reasoning tasks. Read more about this study
- Who is it for?
- Any ageHealthy volunteers can take part
- Conditions the study is for
- Clinical Decision-making
- What is being tested?
- GPT-4
- Could I receive a placebo?
- No, everyone receives an active treatment
- How many people
- 92 participants
- Where
- 1 siteUnited States
- Study dates
- 28 Dec 2023to 19 Apr 2024
- Phase
- Interventional
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages Any age
- All sexes
- Healthy volunteers accepted
- Child, Adult, Older adult
Who may be able to join
- Participants must be licensed physicians and have completed at least post-graduate year 2 (PGY2) of medical training.
- Training in Internal medicine, family medicine, or emergency medicine.
Who cannot join
- Not currently practicing clinically.
Trial locations and contacts
1 site in 1 country.
About this study
Official title: Management Reasoning With AI Chat Bots
This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional management decision support tools on performance on case-based management reasoning tasks.
Read the detailed description
Artificial intelligence (AI) technologies, specifically advanced large language models like OpenAI's ChatGPT, have the potential to improve medical decision-making. Although ChatGPT-4 was not developed for its use in medical-specific applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultation. However, little is known about how ChatGPT augments the clinical reasoning abilities of clinicians.
Clinical reasoning is a complex process involving pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could potentially help reduce clinician workload and decrease the likelihood of mismanagement. However, ChatGPT-4 was not developed for clinical reasoning nor has it been validated for this purpose. Further, it may be subject to disinformation, including convincing confabulations that may mislead clinicians. If clinicians misuse this tool, it may not improve reasoning and could even cause harm. Therefore, it is important to study how clinicians use large language models to augment clinical reasoning prior to routine incorporation into patient care.
In this study, participants will be randomized to answer clinical management cases with or without access to ChatGPT-4. Each case has multiple components, and the participants will be asked to discuss their reasoning for each component. Answers will be graded by independent reviewers blinded to treatment assignment. A grading rubric was developed for each case by a panel of 4-7 expert discussants. Discussants independently developed a rubric for each case, and then any discrepancies were resolved through multiple rounds of discussions.
- Conditions
- Clinical Decision-making
- Being tested
- GPT-4
- Keywords
- clinical decision support
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
GPT-4Other
OpenAI's GPT-4 large language model with chat interface.
| Study group | Role | What they receive |
|---|---|---|
| GPT-4 | Active comparator | Group will be given access to GPT-4 Receives: Other: GPT-4 |
| Usual Resources | No intervention | Group will not be given access to GPT-4 but will be encouraged to use any resources they wish besides large language models (UpToDate, Dynamed, google, etc). |
What is being measured
The primary outcome is the main question the study is designed to answer.
Management Reasoning
Percent correct (range: 0 to 100) for each case.
Measured: Within one-hour study
Secondary outcomes (1)
Time Spent on Management
Time (in minutes) participants spend per case between the two study arms.
Measured: Within one-hour study
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted17 Jan 2024
- Study start28 Dec 2023
- Primary completion19 Apr 2024
- Study completion19 Apr 2024
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Publications (2)Papers the sponsor has linked to this study.
- Other trials for Clinical Decision-makingStudies of the same condition, open and under way.
- More trials run by Stanford UniversityThe sponsor's other registered studies.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Treatment
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Model description
- The trial will be designed as a randomized, two-arm, single-blind parallel group study.
- Masking
- Single blind
- Who is masked
- Outcomes assessor
- Enrolment
- 92 participants (actual)
Sponsor and investigators
- Lead sponsor
- Stanford University
- Sponsor type
- University, hospital or other
- Collaborators
- Beth Israel Deaconess Medical Center, University of Minnesota
- Responsible party
- Principal investigator, Jonathan Chen, Stanford University
- Investigators
- Jonathan H Chen, MD, PhD, Principal investigator, Stanford UniversityAdam Rodman, MD, Principal investigator, Beth Israel Deaconess Medical CenterAndrew Olson, MD, Principal investigator, University of Minnesota
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT06208423
- Sponsor's protocol ID
- 71319b
- Status last verified
- Sep 2024
- First submitted
- 20 Dec 2023
- Last updated
- 27 Sep 2024
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- No
Publications
Goh E, Gallo RJ, Strong E, Weng Y, Kerman H, Freed JA, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat Med. 2025 Apr;31(4):1233-1238. doi: 10.1038/s41591-024-03456-y. Epub 2025 Feb 5.
PubMed 39910272Goh E, Gallo R, Strong E, Weng Y, Kerman H, Freed J, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial. medRxiv [Preprint]. 2024 Aug 7:2024.08.05.24311485. doi: 10.1101/2024.08.05.24311485.
PubMed 39148822
Study at a glance
From the registry record, last updated by the sponsor on 27 Sep 2024. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Management Reasoning With AI Chat Bots
- ClinicalTrials.gov ID
- NCT06208423
- Acronym / sponsor ID
- 71319b
- Conditions
- Clinical Decision-making
- Keywords
- clinical decision support
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 92 participants (actual)
- Lead sponsor
- Stanford UniversityUniversity, hospital or other
- Collaborators
- Beth Israel Deaconess Medical Center, University of Minnesota
- Study start
- 28 Dec 2023 (actual)
- Primary completion
- 19 Apr 2024 (actual)
- Study completion
- 19 Apr 2024 (actual)
- First posted
- 17 Jan 2024 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional management decision support tools on performance on case-based management reasoning tasks.
Detailed description
Artificial intelligence (AI) technologies, specifically advanced large language models like OpenAI's ChatGPT, have the potential to improve medical decision-making. Although ChatGPT-4 was not developed for its use in medical-specific applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultation. However, little is known about how ChatGPT augments the clinical reasoning abilities of clinicians.
Clinical reasoning is a complex process involving pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could potentially help reduce clinician workload and decrease the likelihood of mismanagement. However, ChatGPT-4 was not developed for clinical reasoning nor has it been validated for this purpose. Further, it may be subject to disinformation, including convincing confabulations that may mislead clinicians. If clinicians misuse this tool, it may not improve reasoning and could even cause harm. Therefore, it is important to study how clinicians use large language models to augment clinical reasoning prior to routine incorporation into patient care.
In this study, participants will be randomized to answer clinical management cases with or without access to ChatGPT-4. Each case has multiple components, and the participants will be asked to discuss their reasoning for each component. Answers will be graded by independent reviewers blinded to treatment assignment. A grading rubric was developed for each case by a panel of 4-7 expert discussants. Discussants independently developed a rubric for each case, and then any discrepancies were resolved through multiple rounds of discussions.
Study design
- Primary purpose
- Treatment
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Masking
- Single blind (outcomes assessor)
- Number of arms
- 2
- Model description
- The trial will be designed as a randomized, two-arm, single-blind parallel group study.
| Arm | Type | Receives |
|---|---|---|
| GPT-4 | Active comparator | Other: GPT-4Group will be given access to GPT-4 |
| Usual Resources | No intervention | –Group will not be given access to GPT-4 but will be encouraged to use any resources they wish besides large language models (UpToDate, Dynamed, google, etc). |
Who may qualify
- Sex
- All
- Healthy volunteers
- Accepted
- Age groups
- Child, Adult, Older adult
Key inclusion criteria (2 listed)
- Participants must be licensed physicians and have completed at least post-graduate year 2 (PGY2) of medical training.
- Training in Internal medicine, family medicine, or emergency medicine.
Key exclusion criteria (1 listed)
- Not currently practicing clinically.
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| GPT-4OpenAI's GPT-4 large language model with chat interface. | Other | GPT-4 |
Primary outcome
Management Reasoning
Time frame: Within one-hour study
Percent correct (range: 0 to 100) for each case.
1 secondary outcome
Time Spent on Management
Time frame: Within one-hour study
Results and safety
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 27 Sep 2024.
- Derived from this study: PubMed 39910272Goh E, Gallo RJ, Strong E, Weng Y, Kerman H, Freed JA, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat Med. 2025 Apr;31(4):1233-1238. doi: 10.1038/s41591-024-03456-y. Epub 2025 Feb 5.
- Derived from this study: PubMed 39148822Goh E, Gallo R, Strong E, Weng Y, Kerman H, Freed J, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial. medRxiv [Preprint]. 2024 Aug 7:2024.08.05.24311485. doi: 10.1101/2024.08.05.24311485.
- Individual participant dataSharing planned: No
Related trials for Clinical Decision-making
Not yet recruiting
A Multidimensional Intervention to Improve Ethical Decision-Making Competence Among Nursing Student Interns: A Randomized Controlled Trial
Ethical Decision-Making Competence in Nursing Student Interns
Active, not recruiting
PRecision Oncology Evidence Development in Cancer Treatment - Liquid
Cancer
Active, not recruiting
A Communication Tool to Improve Communication in the ICU
Serious Injury, Communication
Not yet recruiting
Patient-Reported Outcomes Dashboard With Goal Setting to Support Shared Decision-Making in Inflammatory Arthritis
Arthritis, Psoriatic (PsA), Arthritis, Rheumatoid (RA)
Recruiting
Optimized Recruitment to Drive Equitable Research in Stroke, Empowering Decision-making (ORDERed)
Stroke, Stroke (CVA) or Transient Ischemic Attack
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
Jonathan H Chen, MD, PhD principal investigator
Stanford University
Adam Rodman, MD principal investigator
Beth Israel Deaconess Medical Center
Andrew Olson, MD principal investigator
University of Minnesota
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing GPT-4 in people with Clinical Decision-making, run by Stanford University at 1 site in United States.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 2 inclusion and 1 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 27 Sep 2024. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in United States. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Management Reasoning (time frame: Within one-hour study). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT06208423, which also lists 2 linked publications.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 27 Sep 2024.