Pathologic ProcessesNCT06911645
Diagnostic Reasoning With Customized GPT-4 Model
A study testing Immediate access to customized version of GPT-4 and Access to customized version of GPT-4 following use of conventional resources in people with Pathologic Processes and Disease, run by Stanford University at 1 site in United States.
Trial summary
This study will assess the impact of immediate access to a customized version of GPT-4, a large language model, on performance in case-based diagnostic reasoning tasks. Specifically, it will compare this approach to a two-step process where participants first use traditional diagnostic decision support tools to… Read more about this study
- Who is it for?
- Any ageHealthy volunteers can take part
- Conditions the study is for
- Pathologic Processes, Disease
- What is being tested?
- Immediate access to customized version of GPT-4 and Access to customized version of GPT-4 following use of conventional resources
- Could I receive a placebo?
- No, everyone receives an active treatment
- How many people
- 70 participants
- Where
- 1 siteUnited States
- Study dates
- 16 Dec 2024to 24 Jan 2025
- Phase
- Interventional
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages Any age
- All sexes
- Healthy volunteers accepted
- Child, Adult, Older adult
Who may be able to join
- Participants must be licensed physicians and have completed at least post-graduate year 1 (PGY1) of medical training.
- Training in Internal medicine, family medicine, or emergency medicine.
Who cannot join
- Not currently practicing clinically.
- Participated in one of our previous studies that used the same six diagnostic cases.
Trial locations and contacts
1 site in 1 country.
About this study
Official title: Evaluating the Performance of LLMs and Clinicians in Complex Diagnostic Cases: A Randomized Controlled Trial
This study will assess the impact of immediate access to a customized version of GPT-4, a large language model, on performance in case-based diagnostic reasoning tasks. Specifically, it will compare this approach to a two-step process where participants first use traditional diagnostic decision support tools to support their diagnostic reasoning before gaining access to the customized GPT-4 model.
Read the detailed description
Artificial intelligence (AI) technologies, particularly advanced large language models like OpenAI's ChatGPT, have the potential to enhance medical decision-making. While ChatGPT-4 was not specifically designed for medical applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultations. However, its impact on clinicians' diagnostic reasoning remains largely unknown.
Clinical reasoning is a complex process that involves pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could help reduce clinician workload and decrease the likelihood of missed diagnoses. However, ChatGPT-4 was neither developed nor validated for diagnostic reasoning, and it may produce misleading information, including plausible but incorrect conclusions that could misguide clinicians. If not used appropriately, it may fail to improve-and could even hinder-clinical decision-making. Therefore, it is essential to study how clinicians use large language models to support clinical reasoning before integrating them into routine patient care.
This study will examine how immediate access to a customized version of ChatGPT-4 impacts performance on case-based diagnostic reasoning tasks, compared to a stepwise approach. In the stepwise approach, participants will first use traditional diagnostic decision support tools to support their case reasoning before interacting with a customized ChatGPT-4 model, at which point they will have the opportunity to revise their initial answers.
Participants will be randomized into different study arms and will respond to diagnostic cases by providing three differential diagnoses, along with supporting and opposing findings for each. They will also identify their top diagnosis and propose next diagnostic steps. Independent reviewers, blinded to treatment assignment, will evaluate their responses.
- Conditions
- Pathologic Processes, Disease
- Being tested
- Immediate access to customized version of GPT-4, Access to customized version of GPT-4 following use of conventional resources
- Keywords
- Computer-assisted diagnosis, Large language models, Clinical reasoning
- Medical subject headings
- Pathologic Processes, Disease
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
Immediate access to customized version of GPT-4Other
Group is given immediate access to a customized version of GPT-4 to support their diagnostic reasoning for each case.
Access to customized version of GPT-4 following use of conventional resourcesOther
Group is first encouraged to reason through diagnostic cases with the support of conventional resources. After they submit a case's answers they are then given access to a customized version of GPT-4 and have the opportunity to change their initial answers.
| Study group | Role | What they receive |
|---|---|---|
| Immediate access to customized version of GPT-4 | Active comparator | Group will be encouraged to immediately use a customized version of GPT-4. Receives: Other: Immediate access to customized version of GPT-4 |
| Conventional resources first, then granted access to customized version of GPT-4. | Active comparator | Group will be encouraged to first use any resources they wish besides large language models (UpToDate, Pubmed, google, etc) and then will be granted access to a customized version of GPT-4. Receives: Other: Access to customized version of GPT-4 following use of conventional resources |
What is being measured
The primary outcome is the main question the study is designed to answer.
Diagnostic reasoning
The primary outcome will be the percentage of correct responses per case (range: 0 to 100). For each case, participants will be asked to provide their top three differential diagnoses, along with supporting and opposing findings for each. They will receive 1 point for each plausible diagnosis. Supporting and opposing findings will be graded based on correctness, with 1 point for a partially correct response and 2 points for a completely correct response. Participants will then select their top diagnosis, earning 1 point for a reasonable choice and 2 points for the most accurate diagnosis. Finally, they will list up to three next steps for further patient evaluation, with 1 point awarded for a partially correct response and 2 points for a completely correct response. The primary outcome will be analyzed at the case level, comparing performance between the randomized study groups.
Measured: Through study completion, an average of 6 months
Secondary outcomes (5)
Time Spent Per Case
The investigators will compare the average time (in minutes) participants spend on each case across the two study arms.
Measured: Through study completion, an average of 6 months
Prompt frequency
The investigators will compare the frequency of participant prompts to the customized GPT-4 model between the two study groups.
Measured: Through study completion, an average of 6 months
Sentiment
The investigators will compare the tone and sentiment of participant prompts to the customized GPT-4 model across the two study groups. The investigators will create a qualitative coding system to categorize the nature of the participants' prompts.
Measured: Through study completion, an average of 6 months
Participant Perceptions of AI in Clinical Reasoning
This outcome would be assessed in both study arms and would encompass changes in attitudes, confidence, and willingness to use AI diagnostic tools before and after being exposed to the customized tool. We will assess the number of participants who were open to using AI to help with complex clinical reasoning (pre- and post-quiz), if they enjoyed working with the AI diagnostic tool, if they felt like the tool provided a valuable collaborative experience for clinical reasoning, if seeing the AI diagnostic tool's recommendations increased their confidence in their differential diagnoses, and if they would use an AI diagnostic tool like the one in this study in their daily job. These will be evaluated on a Likert scale ranking from strongly disagree to strongly agree.
Measured: Through study completion, an average of 6 months
Customized GPT-4's diagnostic reasoning
The customized GPT-4's 'independent' diagnoses will be assessed for accuracy. The outcome will be the percentage of correct responses per case (range: 0 to 100). For each case, the meta-prompt directs the customized GPT-4 to provide its top three differential diagnoses, along with supporting and opposing findings for each, a final diagnosis, and next steps. The customized GPT-4 will receive 1 point for each plausible diagnosis. Supporting and opposing findings will be graded based on correctness, with 1 point for a partially correct response and 2 points for a completely correct response. Its top diagnosis will earn 1 point for a reasonable choice and 2 points for the most accurate diagnosis. Finally, it will list up to three next steps for further patient evaluation, with 1 point awarded for a partially correct response and 2 points for a completely correct response. The outcome will be analyzed at the case level, comparing performance with the randomized study groups' scores.
Measured: Through study completion, an average of 6 months
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted4 Apr 2025
- Study start16 Dec 2024
- Primary completion24 Jan 2025
- Study completion24 Jan 2025
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Publications (1)Papers the sponsor has linked to this study.
- Other trials for Pathologic ProcessesStudies of the same condition, open and under way.
- More trials run by Stanford UniversityThe sponsor's other registered studies.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Diagnosis
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Model description
- The trial will be designed as a randomized, two-arm, single-blind parallel group study.
- Masking
- Single blind
- Who is masked
- Outcomes assessor
- Enrolment
- 70 participants (actual)
Sponsor and investigators
- Lead sponsor
- Stanford University
- Sponsor type
- University, hospital or other
- Collaborators
- Beth Israel Deaconess Medical Center
- Responsible party
- Principal investigator, Jonathan Chen, Stanford University
- Investigators
- Jonathan H Chen, MD, PhD, Principal investigator, Stanford University
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT06911645
- Sponsor's protocol ID
- 71319c
- Status last verified
- Mar 2025
- First submitted
- 11 Feb 2025
- Last updated
- 4 Apr 2025
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- No
Publications
Everett SS, Bunning BJ, Jain P, Lopez I, Agarwal A, Desai M, Gallo R, Goh E, Kadiyala VB, Kanjee Z, Koshy JM, Olson A, Rodman A, Schulman K, Strong E, Chen JH, Horvitz E. From tool to teammate in a randomized controlled trial of clinician-AI collaborative workflows for diagnosis. NPJ Digit Med. 2026 Mar 18;9(1):409. doi: 10.1038/s41746-026-02545-1.
PubMed 41851268
Study at a glance
From the registry record, last updated by the sponsor on 4 Apr 2025. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Evaluating the Performance of LLMs and Clinicians in Complex Diagnostic Cases: A Randomized Controlled Trial
- ClinicalTrials.gov ID
- NCT06911645
- Acronym / sponsor ID
- 71319c
- Conditions
- Pathologic Processes, Disease
- Keywords
- Computer-assisted diagnosis, Large language models, Clinical reasoning
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 70 participants (actual)
- Lead sponsor
- Stanford UniversityUniversity, hospital or other
- Collaborators
- Beth Israel Deaconess Medical Center
- Study start
- 16 Dec 2024 (actual)
- Primary completion
- 24 Jan 2025 (actual)
- Study completion
- 24 Jan 2025 (actual)
- First posted
- 4 Apr 2025 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
This study will assess the impact of immediate access to a customized version of GPT-4, a large language model, on performance in case-based diagnostic reasoning tasks. Specifically, it will compare this approach to a two-step process where participants first use traditional diagnostic decision support tools to support their diagnostic reasoning before gaining access to the customized GPT-4 model.
Detailed description
Artificial intelligence (AI) technologies, particularly advanced large language models like OpenAI's ChatGPT, have the potential to enhance medical decision-making. While ChatGPT-4 was not specifically designed for medical applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultations. However, its impact on clinicians' diagnostic reasoning remains largely unknown.
Clinical reasoning is a complex process that involves pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could help reduce clinician workload and decrease the likelihood of missed diagnoses. However, ChatGPT-4 was neither developed nor validated for diagnostic reasoning, and it may produce misleading information, including plausible but incorrect conclusions that could misguide clinicians. If not used appropriately, it may fail to improve-and could even hinder-clinical decision-making. Therefore, it is essential to study how clinicians use large language models to support clinical reasoning before integrating them into routine patient care.
This study will examine how immediate access to a customized version of ChatGPT-4 impacts performance on case-based diagnostic reasoning tasks, compared to a stepwise approach. In the stepwise approach, participants will first use traditional diagnostic decision support tools to support their case reasoning before interacting with a customized ChatGPT-4 model, at which point they will have the opportunity to revise their initial answers.
Participants will be randomized into different study arms and will respond to diagnostic cases by providing three differential diagnoses, along with supporting and opposing findings for each. They will also identify their top diagnosis and propose next diagnostic steps. Independent reviewers, blinded to treatment assignment, will evaluate their responses.
Study design
- Primary purpose
- Diagnosis
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Masking
- Single blind (outcomes assessor)
- Number of arms
- 2
- Model description
- The trial will be designed as a randomized, two-arm, single-blind parallel group study.
| Arm | Type | Receives |
|---|---|---|
| Immediate access to customized version of GPT-4 | Active comparator | Other: Immediate access to customized version of GPT-4Group will be encouraged to immediately use a customized version of GPT-4. |
| Conventional resources first, then granted access to customized version of GPT-4. | Active comparator | Other: Access to customized version of GPT-4 following use of conventional resourcesGroup will be encouraged to first use any resources they wish besides large language models (UpToDate, Pubmed, google, etc) and then will be granted access to a customized version of GPT-4. |
Who may qualify
- Sex
- All
- Healthy volunteers
- Accepted
- Age groups
- Child, Adult, Older adult
Key inclusion criteria (2 listed)
- Participants must be licensed physicians and have completed at least post-graduate year 1 (PGY1) of medical training.
- Training in Internal medicine, family medicine, or emergency medicine.
Key exclusion criteria (2 listed)
- Not currently practicing clinically.
- Participated in one of our previous studies that used the same six diagnostic cases.
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| Immediate access to customized version of GPT-4Group is given immediate access to a customized version of GPT-4 to support their diagnostic reasoning for each case. | Other | Immediate access to customized version of GPT-4 |
| Access to customized version of GPT-4 following use of conventional resourcesGroup is first encouraged to reason through diagnostic cases with the support of conventional resources. After they submit a case's answers they are then given access to a customized version of GPT-4 and have the opportunity to change their initial answers. | Other | Conventional resources first, then granted access to customized version of GPT-4. |
Primary outcome
Diagnostic reasoning
Time frame: Through study completion, an average of 6 months
The primary outcome will be the percentage of correct responses per case (range: 0 to 100). For each case, participants will be asked to provide their top three differential diagnoses, along with supporting and opposing findings for each. They will receive 1 point for each plausible diagnosis. Supporting and opposing findings will be graded based on correctness, with 1 point for a partially correct response and 2 points for a completely correct response. Participants will then select their top diagnosis, earning 1 point for a reasonable choice and 2 points for the most accurate diagnosis. Finally, they will list up to three next steps for further patient evaluation, with 1 point awarded for a partially correct response and 2 points for a completely correct response. The primary outcome will be analyzed at the case level, comparing performance between the randomized study groups.
5 secondary outcomes
Time Spent Per Case
Time frame: Through study completion, an average of 6 months
Prompt frequency
Time frame: Through study completion, an average of 6 months
Sentiment
Time frame: Through study completion, an average of 6 months
Participant Perceptions of AI in Clinical Reasoning
Time frame: Through study completion, an average of 6 months
Customized GPT-4's diagnostic reasoning
Time frame: Through study completion, an average of 6 months
Results and safety
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 4 Apr 2025.
- Derived from this study: PubMed 41851268Everett SS, Bunning BJ, Jain P, Lopez I, Agarwal A, Desai M, Gallo R, Goh E, Kadiyala VB, Kanjee Z, Koshy JM, Olson A, Rodman A, Schulman K, Strong E, Chen JH, Horvitz E. From tool to teammate in a randomized controlled trial of clinician-AI collaborative workflows for diagnosis. NPJ Digit Med. 2026 Mar 18;9(1):409. doi: 10.1038/s41746-026-02545-1.
- Individual participant dataSharing planned: No
Related trials for Pathologic Processes
RecruitingPhase 3
A Non-inferiority Pharmacokinetic and Safety/Tolerability Study of Two Different Doses of Weekly SC Alpha1-PI 15% Compared With Corresponding Standard IV Alpha1-PI in Participants With Alpha1-Antitrypsin Deficiency (AATD)
Alpha 1 Antitrypsin Deficiency
Recruiting
Safety and Effectiveness of the Sphere-9™ Catheter and Affera™ Ablation System for the Treatment of Ventricular Tachycardia (Sphere VT)
Ventricular Tachycardia
RecruitingPhase 1
A Study of AIR-001 in Adults With Alpha-1 Antitrypsin Deficiency (AATD)
Alpha 1 Antitrypsin Deficiency
Active, not recruitingPhase 3
Pharmacokinetic Study of QL2107 Versus Keytruda® for Adjuvant Therapy of Non-Small Cell Lung Cancer (NSCLC)
Carcinoma, Non-Small-Cell Lung
RecruitingObservational study
This Study is the Clinical Validation Study for Validation for the Digital Pathology System 'SGi 1.0 + EPW 1.0 + Compatible Clinical Displays'
Pathological Processes
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
Jonathan H Chen, MD, PhD principal investigator
Stanford University
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing Immediate access to customized version of GPT-4 and Access to customized version of GPT-4 following use of conventional resources in people with Pathologic Processes and Disease, run by Stanford University at 1 site in United States.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 2 inclusion and 2 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 4 Apr 2025. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in United States. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Diagnostic reasoning (time frame: Through study completion, an average of 6 months). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT06911645, which also lists 1 linked publication.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 4 Apr 2025; AzTrials last refreshed its copy on 28 Oct 2026.