Large Language ModelsPRECISENCT07037940

Physician Response Evaluation With Contextual Insights vs. Standard Engines - Artificial Intelligence RAG vs LLM Clinical Decision Support

A study testing OpenEvidence and GPT-4 in people with Large Language Models, run by Montefiore Medical Center at 2 sites in United States.

Completed

Finished. The study has ended and is not enrolling. Results may be posted on the official record.

Official record

Search all trials

Trial summary

Who is it for?: Ages 25 years and olderHealthy volunteers can take part
Conditions the study is for: Large Language Models
What is being tested?: OpenEvidence and GPT-4
Could I receive a placebo?: No, everyone receives an active treatment
How many people: 41 participants
Where: 2 sitesUnited States
Study dates: 3 Jul 2025to 30 Dec 2025
Phase: Interventional
Are there results?: None posted in the registry

Key requirements

Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.

Ages 25 years and older
All sexes
Healthy volunteers accepted
Adult, Older adult

Who may be able to join

Internal medicine residents
Internal medicine attending physicians

Who cannot join

Not meeting Inclusion Criteria

Trial locations and contacts

2 sites in 1 country.

United States2 sites

Harvard Beth Israel Deaconess Medical Center
Boston, Massachusetts, 02215
MontefioreMC
The Bronx, New York, 10467

Other trials in United States

About this study

Official title: Comparing OpenEvidence and GPT-4 for Diagnostic Reasoning and Management Decisions: A Randomized Trial

Clinical decision support tools powered by artificial intelligence (AI) are being rapidly integrated into medical practice. Two leading systems currently available to clinicians are OpenEvidence, which uses retrieval-augmented generation to access medical literature, and GPT-4, a large language model. While both tools show promise, their relative effectiveness in supporting clinical decision-making has not been directly compared. This study aims to evaluate how these tools influence diagnostic reasoning and management decisions among internal medicine physicians.

Read the detailed description

Internal medicine attendings and residents are invited to participate in a study investigating how physicians using a RAG-based LLM (OpenEvidence) perform compared to those using a standard general-purpose LLM (ChatGPT) on both diagnostic reasoning and complex management decisions. As AI tools increasingly enter clinical practice, evidence is needed about which approaches best support physician decision-making. This study will help determine if specialized medical knowledge retrieval systems (OpenEvidence) provide advantages over general AI assistants (ChatGPT) when solving real clinical cases.

Participants will complete one 90-minute Zoom session where clinical cases derived from real, de-identified patient encounters will be solved. Participants will be randomly assigned to use either OpenEvidence or ChatGPT and all responses evaluated by blinded scorers using a validated rubric.

Conditions: Large Language Models
Being tested: OpenEvidence, GPT-4

What participants receive

What participants receive. In a randomised study each person is assigned to one group by chance.

OpenEvidenceOther
Medical information platform which uses retrieval-augmented generation to access medical literature
GPT-4Other
Also called ChatGPT
A chatbot application developed that uses GPT-4, a large language model, to engage in conversational interactions with users.

Study groups in NCT07037940
Study group	Role	What they receive
OpenEvidence	Active comparator	Participants in this arm will use OpenEvidence as their research tool Receives: Other: OpenEvidence
ChatGPT	Active comparator	Patients in this arm will use Chat-GPT as their research tool Receives: Other: GPT-4

What is being measured

The primary outcome is the main question the study is designed to answer.

Clinical Reasoning Performance as determined by Rater Scores
Clinical reasoning performance will be evaluated based upon the accuracy of the rater scores to responses to the surveys administered. Six blinded, trained independent raters will independently score each participant's response using a validated scoring rubric. Possible response scores can range from 0-100% with higher scores indicating increased clinical reasoning performance. Results for each assessment will be summarized by study arm using basic descriptive statistics and analyzed using mixed-effects models to account for within-subject correlation and between-subject factors.
Measured: 15-minutes upon completion of cases, up to approximately 90 minutes total

Secondary outcomes (2)

Time efficiency
Time efficiency will be assessed based on the amount of time it takes for participants to complete the surveys. Each survey will automatically be time stamped to record the amount of time needed for each participant to answer each case. Results for the virtual session will be summarized by study arm using basic descriptive statistics and analyzed.
Measured: Up to approximately 75 minutes
Decision confidence
Decision confidence will be determined by asking participants to assess the level of confidence in survey answers using a scale ranging from 1-5 (1 being least confident, 5 being most confident) such that higher scores are associated with increased confidence in responses. Scores will be summarized by study arm using basic descriptive statistics.
Measured: 15-minutes upon completion of cases, up to approximately 90 minutes total

Timeline

From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.

3 Jul 202530 Dec 2025

First posted26 Jun 2025
Study start3 Jul 2025
Primary completion30 Dec 2025
Study completion30 Dec 2025

Resources

Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
Publications (6)Papers the sponsor has linked to this study.
Other trials for Large Language ModelsStudies of the same condition, open and under way.
What the trial phases meanHow a treatment moves from first tests in people to approval.

More details

For doctors and researchers: how the study is designed, who runs it, and its registry IDs.

Study design

Study type: Interventional
Phase: Phase N/A
Primary purpose: Other
Allocation: Randomised
Intervention model: Parallel groups
Masking: Single blind
Who is masked: Outcomes assessor
Enrolment: 41 participants (actual)

IDs, dates and publications

ClinicalTrials.gov ID: NCT07037940
Sponsor's protocol ID: 2025-16599
Other IDs: 2025
Acronym: PRECISE
Status last verified: Aug 2026
First submitted: 17 Jun 2025
Last updated: 11 Aug 2026
Results posted: No
Data monitoring committee: No
FDA-regulated drug: No
FDA-regulated device: No
Individual data sharing: No

Publications

Cabral S, Restrepo D, Kanjee Z, Wilson P, Crowe B, Abdulnour RE, Rodman A. Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physicians. JAMA Intern Med. 2024 May 1;184(5):581-583. doi: 10.1001/jamainternmed.2024.0295.
PubMed 38557971
Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, Scales N, Tanwani A, Cole-Lewis H, Pfohl S, Payne P, Seneviratne M, Gamble P, Kelly C, Babiker A, Scharli N, Chowdhery A, Mansfield P, Demner-Fushman D, Aguera Y Arcas B, Webster D, Corrado GS, Matias Y, Chou K, Gottweis J, Tomasev N, Liu Y, Rajkomar A, Barral J, Semturs C, Karthikesalingam A, Natarajan V. Large language models encode clinical knowledge. Nature. 2023 Aug;620(7972):172-180. doi: 10.1038/s41586-023-06291-2. Epub 2023 Jul 12.
PubMed 37438534
Strong E, DiGiammarino A, Weng Y, Kumar A, Hosamani P, Hom J, Chen JH. Chatbot vs Medical Student Performance on Free-Response Clinical Reasoning Examinations. JAMA Intern Med. 2023 Sep 1;183(9):1028-1030. doi: 10.1001/jamainternmed.2023.2909.
PubMed 37459090
Schaye V, Miller L, Kudlowitz D, Chun J, Burk-Rafel J, Cocks P, Guzman B, Aphinyanaphongs Y, Marin M. Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes: a Shared Mental Model for Feedback. J Gen Intern Med. 2022 Feb;37(3):507-512. doi: 10.1007/s11606-021-06805-6. Epub 2021 May 4.
PubMed 33945113
Goh E, Gallo R, Hom J, Strong E, Weng Y, Kerman H, Cool JA, Kanjee Z, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Rodman A, Chen JH. Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial. JAMA Netw Open. 2024 Oct 1;7(10):e2440969. doi: 10.1001/jamanetworkopen.2024.40969.
PubMed 39466245
Goh E, Gallo RJ, Strong E, Weng Y, Kerman H, Freed JA, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat Med. 2025 Apr;31(4):1233-1238. doi: 10.1038/s41591-024-03456-y. Epub 2025 Feb 5.
PubMed 39910272

Want to know more?

The sponsor has not listed a contact. The official record may have more details.

Talk to your doctor first and take this page with you. Only the study team can confirm whether you can join.

Questions to ask the study team

What is the study trying to find out?
What would I need to do, and how often would I visit?
Could I receive a placebo instead of the treatment?
What are the possible risks and benefits for me?
Will it cost me anything? Are travel or time paid for?
Can I keep seeing my own doctor during the study?
What happens when the study ends?

See locations

Not yet recruitingObservational study
Diagnostic Accuracy of Large Language Models for Peri-Implant Marginal Bone Loss on Panoramic Radiographs
Peri-implantitis, Alveolar Bone Loss
Recruiting
Designing a Large Language Model Architecture for Shared Decision Making and Informed Intentions
Shared Decision Making, Health Literacy
Not yet recruiting
CTEPH Differential Diagnosis Assisted by Enhanced Large Language Model Support
Chronic Thromboembolic Pulmonary Hypertension (CTEPH), Acute Pulmonary Embolism (PE)
Recruiting
Vision-Language-Based Large-Scale Model for Emergency Screening
Acute Thoraco-Abdominal Emergency Lesions
Not yet recruiting
Conversational AI for Adult Flu Vaccination Decisions
Influenza, Human, Vaccine Hesitancy

All Large Language Models trials

Study at a glance

From the registry record, last updated by the sponsor on 11 Aug 2026. Recruitment status is as the sponsor reported it: confirm with the study team.

Official title: Comparing OpenEvidence and GPT-4 for Diagnostic Reasoning and Management Decisions: A Randomized Trial
ClinicalTrials.gov ID: NCT07037940
Acronym / sponsor ID: PRECISE / 2025-16599
Conditions: Large Language Models
Study type: Interventional
Recruitment status: Completed
Enrolment: 41 participants (actual)
Lead sponsor: Montefiore Medical CenterUniversity, hospital or other
Collaborators: Beth Israel Deaconess Medical Center
Study start: 3 Jul 2025 (actual)
Primary completion: 30 Dec 2025 (actual)
Study completion: 30 Dec 2025 (actual)
First posted: 26 Jun 2025 (actual)
Results: Not posted in the source record

Clinical summary

The research question as the sponsor registered it. This describes what the study plans to test, not what it found.

Detailed description

Study design

Primary purpose: Other
Allocation: Randomised
Intervention model: Parallel groups
Masking: Single blind (outcomes assessor)
Number of arms: 2

Arm	Type	Receives
OpenEvidence	Active comparator	Other: OpenEvidenceParticipants in this arm will use OpenEvidence as their research tool
ChatGPT	Active comparator	Other: GPT-4Patients in this arm will use Chat-GPT as their research tool

Who may qualify

Ages: 25 years and older
Sex: All
Healthy volunteers: Accepted
Age groups: Adult, Older adult

Key inclusion criteria (2 listed)

Internal medicine residents
Internal medicine attending physicians

Key exclusion criteria (1 listed)

Not meeting Inclusion Criteria

Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.

Interventions

Intervention	Type	Arms
OpenEvidenceMedical information platform which uses retrieval-augmented generation to access medical literature	Other	OpenEvidence
GPT-4Also: ChatGPTA chatbot application developed that uses GPT-4, a large language model, to engage in conversational interactions with users.	Other	ChatGPT

Primary outcome

Clinical Reasoning Performance as determined by Rater Scores
Time frame: 15-minutes upon completion of cases, up to approximately 90 minutes total
Clinical reasoning performance will be evaluated based upon the accuracy of the rater scores to responses to the surveys administered. Six blinded, trained independent raters will independently score each participant's response using a validated scoring rubric. Possible response scores can range from 0-100% with higher scores indicating increased clinical reasoning performance. Results for each assessment will be summarized by study arm using basic descriptive statistics and analyzed using mixed-effects models to account for within-subject correlation and between-subject factors.

2 secondary outcomes

Time efficiency
Time frame: Up to approximately 75 minutes
Decision confidence
Time frame: 15-minutes upon completion of cases, up to approximately 90 minutes total

Results and safety

The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.

Research links

Official record on ClinicalTrials.govLast updated by the sponsor 11 Aug 2026.
Background: PubMed 38557971Cabral S, Restrepo D, Kanjee Z, Wilson P, Crowe B, Abdulnour RE, Rodman A. Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physicians. JAMA Intern Med. 2024 May 1;184(5):581-583. doi: 10.1001/jamainternmed.2024.0295.
Background: PubMed 37438534Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, Scales N, Tanwani A, Cole-Lewis H, Pfohl S, Payne P, Seneviratne M, Gamble P, Kelly C, Babiker A, Scharli N, Chowdhery A, Mansfield P, Demner-Fushman D, Aguera Y Arcas B, Webster D, Corrado GS, Matias Y, Chou K, Gottweis J, Tomasev N, Liu Y, Rajkomar A, Barral J, Semturs C, Karthikesalingam A, Natarajan V. Large language models encode clinical knowledge. Nature. 2023 Aug;620(7972):172-180. doi: 10.1038/s41586-023-06291-2. Epub 2023 Jul 12.
Background: PubMed 37459090Strong E, DiGiammarino A, Weng Y, Kumar A, Hosamani P, Hom J, Chen JH. Chatbot vs Medical Student Performance on Free-Response Clinical Reasoning Examinations. JAMA Intern Med. 2023 Sep 1;183(9):1028-1030. doi: 10.1001/jamainternmed.2023.2909.
Background: PubMed 33945113Schaye V, Miller L, Kudlowitz D, Chun J, Burk-Rafel J, Cocks P, Guzman B, Aphinyanaphongs Y, Marin M. Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes: a Shared Mental Model for Feedback. J Gen Intern Med. 2022 Feb;37(3):507-512. doi: 10.1007/s11606-021-06805-6. Epub 2021 May 4.
Background: PubMed 39466245Goh E, Gallo R, Hom J, Strong E, Weng Y, Kerman H, Cool JA, Kanjee Z, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Rodman A, Chen JH. Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial. JAMA Netw Open. 2024 Oct 1;7(10):e2440969. doi: 10.1001/jamanetworkopen.2024.40969.
Background: PubMed 39910272Goh E, Gallo RJ, Strong E, Weng Y, Kerman H, Freed JA, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat Med. 2025 Apr;31(4):1233-1238. doi: 10.1038/s41591-024-03456-y. Epub 2025 Feb 5.
Individual participant dataSharing planned: No

Related trials for Large Language Models

Not yet recruitingObservational study
Diagnostic Accuracy of Large Language Models for Peri-Implant Marginal Bone Loss on Panoramic Radiographs
Peri-implantitis, Alveolar Bone Loss
Recruiting
Designing a Large Language Model Architecture for Shared Decision Making and Informed Intentions
Shared Decision Making, Health Literacy
Not yet recruiting
CTEPH Differential Diagnosis Assisted by Enhanced Large Language Model Support
Chronic Thromboembolic Pulmonary Hypertension (CTEPH), Acute Pulmonary Embolism (PE)
Recruiting
Vision-Language-Based Large-Scale Model for Emergency Screening
Acute Thoraco-Abdominal Emergency Lesions
Not yet recruiting
Conversational AI for Adult Flu Vaccination Decisions
Influenza, Human, Vaccine Hesitancy

Referral and contact

AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.

Shitij Arora, MD principal investigator

Montefiore Medical Center

2 sites in 1 country: see every site and its status.

Questions about this trial

What is this clinical trial studying?

A study testing OpenEvidence and GPT-4 in people with Large Language Models, run by Montefiore Medical Center at 2 sites in United States.

The full registered description is in the About this study section and the official record.

Who may be eligible to take part?

The sponsor lists 2 inclusion and 1 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.

Is this clinical trial currently recruiting?

The record says: finished, as of the sponsor's last update on 11 Aug 2026. Status can change between updates, so confirm with the study contact before making plans.

Where is the study taking place?

The record lists 2 sites in United States. Whether a site is taking part right now should be confirmed with that site. See every site.

What are the possible benefits and risks?

They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.

Are results available for this study?

The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.

How can a doctor refer a patient or contact the research team?

The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.

What does the primary outcome measure?

Clinical Reasoning Performance as determined by Rater Scores (time frame: 15-minutes upon completion of cases, up to approximately 90 minutes total). It is what the study was designed to measure, not a finding.

Where can I read the original study information?

On the ClinicalTrials.gov record for NCT07037940, which also lists 6 linked publications.

Does a listing mean the treatment is approved or proven to work?

No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.

How often is this information updated?

AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 11 Aug 2026.