Artificial Intelligence (AI)EDuCATeNCT06921447
Evaluating Decision-making Using ChatGPT-4 Among Trainees in Surgery
An observational study of Artificial Intelligence (AI) and Training, run by Ospedali Riuniti Trieste at 1 site in Italy.
Trial summary
This study aims to assess whether ChatGPT-4 can support surgical trainees in clinical decision-making. By comparing the performance of ChatGPT-4 with junior residents, senior residents, and attending surgeons on standardized clinical scenarios, the study seeks to understand the potential role of large language models… Read more about this study
- Who is it for?
- Any age
- Conditions the study is for
- Artificial Intelligence (AI), Training, Surgery
- What is being tested?
- Clinical Cases
- How many people
- 35 participants
- Where
- 1 siteItaly
- Study dates
- 10 Apr 2025to 30 Apr 2025
- Phase
- Observational
- Are there results?
- Not yet: the study is under way
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages Any age
- All sexes
- No healthy volunteers
- Child, Adult, Older adult
Study population: The study population will consist of general surgery trainees and faculty members from a single academic surgical department. Participants will be stratified into three groups based on their level of training and experience: Junior Residents: Postgraduate Year (PGY) 1-2; Senior Residents: PGY 3 and above; Attending Surgeons: Board-certified general surgeons with independent clinical practice
Who may be able to join
- Actively enrolled or employed in the general surgery residency or department at the participating institution
- Willingness to participate and complete all clinical case scenarios
- Consent to participate in the study
Who cannot join
- Incomplete responses
- Use of external assistance (e.g., internet search, AI tools) when answering scenarios, as self-reported in instructions
Trial locations and contacts
1 site in 1 country. Contact the study team or a site near you to ask about taking part.
Italy1 site
University of Trieste
Trieste
Manuela Mastronardi, 0403994152, manuela.mastronardi@gmail.com
Silvia Palmisano
About this study
Official title: Evaluating ChatGPT-4 as a Decision-Making Support Tool for Surgical Trainees
This study aims to assess whether ChatGPT-4 can support surgical trainees in clinical decision-making. By comparing the performance of ChatGPT-4 with junior residents, senior residents, and attending surgeons on standardized clinical scenarios, the study seeks to understand the potential role of large language models in surgical education. The ultimate goal is to evaluate whether ChatGPT-4 can be safely integrated as a supplementary educational tool to aid junior residents in developing critical thinking and surgical judgment.
Read the detailed description
Background:
Artificial Intelligence (AI) is rapidly transforming the medical landscape, offering new possibilities in education, diagnostics, and decision support. In surgery, clinical decision-making is a core competency developed progressively through training. ChatGPT-4, a state-of-the-art large language model developed by OpenAI, has demonstrated competence in handling medical queries and clinical reasoning tasks. However, its performance in complex surgical decision-making compared to human trainees remains largely unexplored.
Objective:
The EDuCATe study aims to evaluate the accuracy and reliability of ChatGPT-4's responses to clinical scenarios involving general surgery cases. Specifically, the study compares the model's performance to that of junior residents, senior residents, and attending surgeons to understand if ChatGPT-4 can serve as a safe and effective educational tool for surgical trainees.
Methods:
Seven clinical scenarios will be constructed using real anonymized patient data representing common general surgery conditions. Each case will be presented step-by-step, mimicking the clinical decision-making process. Participants will answer a question related to treatment choice.
Participants will include junior residents (PGY1-2), senior residents (PGY3+), and attending surgeons from a single surgical department. ChatGPT-4 will be prompted with the same scenarios. All participants will be instructed to complete the cases without using external resources such as AI tools or internet searches, relying solely on their clinical knowledge.
Statistical analysis will compare performance across groups using non-parametric tests (e.g., Wilcoxon rank sum).
Expected Outcomes:
The study hypothesizes that ChatGPT-4 will perform at a level comparable to senior residents or attending surgeons and outperform junior residents in decision-making. If confirmed, these results could support the safe use of ChatGPT-4 as a training aid for junior surgical residents, potentially improving educational outcomes and clinical reasoning skills.
Significance:
This study will provide novel insight into the role of AI in surgical education. By rigorously comparing ChatGPT-4's decision-making capabilities to that of human surgeons at various levels, the study hopes to define its utility, limitations, and appropriate use in residency training programs.
- Conditions
- Artificial Intelligence (AI), Training, Surgery
- Being tested
- Clinical Cases
- Keywords
- artificial intelligence, surgery, training, oncology, emergency
- Medical subject headings
- Neoplasms, Emergencies
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
Clinical CasesOther
Seven clinical cases that have to be analysed
| Study group | Role | What they receive |
|---|---|---|
| Residents | Junior and Senior Residents in General Surgery Receives: Other: Clinical Cases | |
| Consultants | Senior General Surgeons Receives: Other: Clinical Cases | |
| ChatGPT 4 | Artificial Intelligence System Receives: Other: Clinical Cases |
What is being measured
The primary outcome is the main question the study is designed to answer.
Proportion of correct responses
Binary outcome (correct vs. incorrect decision)
Measured: Baseline
Secondary outcomes (3)
Comparison of accuracy across experience levels
Proportion of correct responses by group: Junior residents, Senior residents, Attending surgeons, ChatGPT-4
Measured: Baseline
Confidence level
Participants and ChatGPT are asked to rate how confident they feel in their answer (1-5 Likert scale, where 1 means no confident and 5 very confident)
Measured: Baseline
Percentage of use of AI for clinical cases evaluation
Participants are asked if they use or not ChatGPT in their clinical activity
Measured: Baseline
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted10 Apr 2025
- Study start10 Apr 2025 (estimated)
- Primary completion25 Apr 2025 (estimated)
- Study completion30 Apr 2025 (estimated)
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Other trials for Artificial Intelligence (AI)Studies of the same condition, open and under way.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Observational
- Observational model
- Cohort
- Time perspective
- Prospective
- Enrolment
- 35 participants (estimated)
Sponsor and investigators
- Lead sponsor
- Ospedali Riuniti Trieste
- Sponsor type
- University, hospital or other
- Responsible party
- Principal investigator, Manuela Mastronardi, Ospedali Riuniti Trieste
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT06921447
- Sponsor's protocol ID
- AI-Surgical training
- Acronym
- EDuCATe
- Status last verified
- Apr 2025
- First submitted
- 2 Apr 2025
- Last updated
- 10 Apr 2025
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- No
Study at a glance
From the registry record, last updated by the sponsor on 10 Apr 2025. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Evaluating ChatGPT-4 as a Decision-Making Support Tool for Surgical Trainees
- ClinicalTrials.gov ID
- NCT06921447
- Acronym / sponsor ID
- EDuCATe / AI-Surgical training
- Conditions
- Artificial Intelligence (AI), Training, Surgery
- Keywords
- artificial intelligence, surgery, training, oncology, emergency
- Study type
- Observational
- Recruitment status
- Not yet recruiting
- Enrolment
- 35 participants (estimated)
- Lead sponsor
- Ospedali Riuniti TriesteUniversity, hospital or other
- Study start
- 10 Apr 2025 (estimated)
- Primary completion
- 25 Apr 2025 (estimated)
- Study completion
- 30 Apr 2025 (estimated)
- First posted
- 10 Apr 2025 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
This study aims to assess whether ChatGPT-4 can support surgical trainees in clinical decision-making. By comparing the performance of ChatGPT-4 with junior residents, senior residents, and attending surgeons on standardized clinical scenarios, the study seeks to understand the potential role of large language models in surgical education. The ultimate goal is to evaluate whether ChatGPT-4 can be safely integrated as a supplementary educational tool to aid junior residents in developing critical thinking and surgical judgment.
Detailed description
Background:
Artificial Intelligence (AI) is rapidly transforming the medical landscape, offering new possibilities in education, diagnostics, and decision support. In surgery, clinical decision-making is a core competency developed progressively through training. ChatGPT-4, a state-of-the-art large language model developed by OpenAI, has demonstrated competence in handling medical queries and clinical reasoning tasks. However, its performance in complex surgical decision-making compared to human trainees remains largely unexplored.
Objective:
The EDuCATe study aims to evaluate the accuracy and reliability of ChatGPT-4's responses to clinical scenarios involving general surgery cases. Specifically, the study compares the model's performance to that of junior residents, senior residents, and attending surgeons to understand if ChatGPT-4 can serve as a safe and effective educational tool for surgical trainees.
Methods:
Seven clinical scenarios will be constructed using real anonymized patient data representing common general surgery conditions. Each case will be presented step-by-step, mimicking the clinical decision-making process. Participants will answer a question related to treatment choice.
Participants will include junior residents (PGY1-2), senior residents (PGY3+), and attending surgeons from a single surgical department. ChatGPT-4 will be prompted with the same scenarios. All participants will be instructed to complete the cases without using external resources such as AI tools or internet searches, relying solely on their clinical knowledge.
Statistical analysis will compare performance across groups using non-parametric tests (e.g., Wilcoxon rank sum).
Expected Outcomes:
The study hypothesizes that ChatGPT-4 will perform at a level comparable to senior residents or attending surgeons and outperform junior residents in decision-making. If confirmed, these results could support the safe use of ChatGPT-4 as a training aid for junior surgical residents, potentially improving educational outcomes and clinical reasoning skills.
Significance:
This study will provide novel insight into the role of AI in surgical education. By rigorously comparing ChatGPT-4's decision-making capabilities to that of human surgeons at various levels, the study hopes to define its utility, limitations, and appropriate use in residency training programs.
Study design
- Observational model
- Cohort
- Time perspective
- Prospective
- Number of arms
- 3
| Arm | Type | Receives |
|---|---|---|
| Residents | – | Other: Clinical CasesJunior and Senior Residents in General Surgery |
| Consultants | – | Other: Clinical CasesSenior General Surgeons |
| ChatGPT 4 | – | Other: Clinical CasesArtificial Intelligence System |
Who may qualify
- Sex
- All
- Healthy volunteers
- Not accepted
- Age groups
- Child, Adult, Older adult
- Study population
- The study population will consist of general surgery trainees and faculty members from a single academic surgical department. Participants will be stratified into three groups based on their level of training and experience: Junior Residents: Postgraduate Year (PGY) 1-2; Senior Residents: PGY 3 and above; Attending Surgeons: Board-certified general surgeons with independent clinical practice
Key inclusion criteria (3 listed)
- Actively enrolled or employed in the general surgery residency or department at the participating institution
- Willingness to participate and complete all clinical case scenarios
- Consent to participate in the study
Key exclusion criteria (2 listed)
- Incomplete responses
- Use of external assistance (e.g., internet search, AI tools) when answering scenarios, as self-reported in instructions
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| Clinical CasesSeven clinical cases that have to be analysed | Other | ChatGPT 4, Consultants, Residents |
Primary outcome
Proportion of correct responses
Time frame: Baseline
Binary outcome (correct vs. incorrect decision)
3 secondary outcomes
Comparison of accuracy across experience levels
Time frame: Baseline
Confidence level
Time frame: Baseline
Percentage of use of AI for clinical cases evaluation
Time frame: Baseline
Results and safety
This study is still under way. Results are not available in the source record yet.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 10 Apr 2025.
- Individual participant dataSharing planned: No
Related trials for Artificial Intelligence (AI)
RecruitingObservational study
Feasibility Study of a Chatbot AI-Driven Preliminary Screening System for Hereditary Cancer Syndromes
Hereditary Cancer Syndromes
Active, not recruiting
AI Ambient Scribe in Psychiatry
Therapeutic Alliance, Empathy
Not yet recruiting
Artificial Intelligence-assisted Uro-Cam Catheter Assessment System for Diagnosing Bladder Cancer
Bladder Cancer
Not yet recruiting
AI-Assisted Systemic Health Risk Assessment in General Ophthalmology
Systemic Health Risk
Not yet recruiting
Impact of Generative AI on University Students' Learning Processes and Critical Thinking
Education, Academic Performance
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part.
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
An observational study of Artificial Intelligence (AI) and Training, run by Ospedali Riuniti Trieste at 1 site in Italy.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 3 inclusion and 2 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: opening soon, as of the sponsor's last update on 10 Apr 2025. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in Italy. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
This study is still under way. Results are not available in the source record yet.
How can a doctor refer a patient or contact the research team?
Contact Manuela Mastronardi (0403994152, manuela.mastronardi@gmail.com), or a site near the patient. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Proportion of correct responses (time frame: Baseline). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 10 Apr 2025.