Emergency MedicineLLM-HEARTNCT07626060
Diagnostic Accuracy of GPT-4o and Claude for HEART Score Calculation in Chest Pain
An observational study of Emergency Medicine and Artificial Intelligence (AI), run by Marmara University Pendik Training and Research Hospital at 1 site in Turkey (Türkiye).
Trial summary
This prospective observational diagnostic accuracy study evaluates whether large language models (LLMs) - GPT-4o (OpenAI, gpt-4o-2024-11-20) and Claude (Anthropic, claude-sonnet-4-6) - can accurately calculate HEART scores from unstructured Turkish clinical notes and predict 30-day major adverse cardiac events (MACE)… Read more about this study
- Who is it for?
- Ages 18 years and older
- Conditions the study is for
- Emergency Medicine, Artificial Intelligence (AI), Artificial Intelligence (AI) in Diagnosis
- What is being tested?
- GPT-4o HEART Score Calculator, Claude HEART Score Calculator and Three-Expert Consensus HEART Score
- How many people
- 690 participants
- Where
- 1 siteTurkey (Türkiye)
- Study dates
- Jun 2026to Jun 2027
- Phase
- Observational
- Are there results?
- Not yet: the study is under way
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages 18 years and older
- All sexes
- No healthy volunteers
- Adult, Older adult
Study population: The study population consists of consecutive adult patients presenting with a chief complaint of non-traumatic chest pain to the emergency department of Marmara University Pendik Training and Research Hospital, a tertiary care academic medical center in Istanbul, Turkey. This target population comprises real-world emergency medicine admissions that require acute coronary syndrome risk stratification and evaluation with the HEART score. It excludes individuals presenting with traumatic pain etiologies or acute ST-elevation myocardial infarction (STEMI) requiring immediate, time-critical reperfusion pathways.
Who may be able to join
- Age >=18 years
- Chief complaint of non-traumatic chest pain at the emergency department
- Written informed consent obtained from the patient or legally authorized representative
- Availability for 30-day follow-up (reachable by telephone and/or actively registered in the e-Nabiz national health database)
Who cannot join
- Traumatic chest pain etiology
- ST-elevation myocardial infarction (STEMI) at presentation requiring immediate reperfusion protocol
- Refusal or subsequent withdrawal of informed consent
- Inability to complete the mandatory 30-day follow-up period
- WITHDRAWAL CRITERIA:
- Patient or representative requests data withdrawal after initial consent
- Administrative identification of retrospective data entry after enrollment
Trial locations and contacts
1 site in 1 country. Contact the study team or a site near you to ask about taking part.
Emre Kudu, associate professor contact
Turkey (Türkiye)1 site, 1 recruiting
- Recruiting
Marmara University Pendik Training and Research Hospital
Istanbul, Istanbul, 34870
Emir ünal, 05327766010, emirunal@gmail.com
Emre Kudu
About this study
Official title: Diagnostic Accuracy of Large Language Models (GPT-4o and Claude) in HEART Score Calculation and 30-Day MACE Prediction in Emergency Department Chest Pain Patients: A Prospective Observational Validation Study Against Three-Expert Consensus
This prospective observational diagnostic accuracy study evaluates whether large language models (LLMs) - GPT-4o (OpenAI, gpt-4o-2024-11-20) and Claude (Anthropic, claude-sonnet-4-6) - can accurately calculate HEART scores from unstructured Turkish clinical notes and predict 30-day major adverse cardiac events (MACE) in emergency department patients presenting with non-traumatic chest pain.
The study will enroll 600 consecutive adult patients. For each patient, the same anonymized data (free-text anamnesis, ECG report text, troponin value, and age) will be independently processed by both LLMs via separate API calls with deterministic settings (temperature=0, JSON format). A three-expert consensus HEART score - derived through blinded independent scoring by three emergency medicine physicians with majority-vote adjudication - serves as the reference standard for agreement analysis. Actual 30-day MACE (all-cause death, AMI Type 1/2/4b, unplanned revascularization) determined via national health database and telephone follow-up serves as the outcome for diagnostic accuracy analysis.
A secondary documentation-quality sub-study will quantify how spontaneously Turkish emergency anamnesis notes capture HEART score parameters.
Read the detailed description
AI SYSTEM SPECIFICATIONS AND PROMPT PROTOCOL Two distinct large language models (LLMs) will be evaluated as index tests: OpenAI GPT-4o (model string: gpt-4o-2024-11-20) and Anthropic Claude (model string: claude-sonnet-4-6). To ensure reproducibility and eliminate stochastic variation, both models will be accessed via standardized API calls using deterministic parameters (temperature = 0, max_tokens = 500, and strict JSON response format). The exact system prompt layout will be locked prior to initialization, and its integrity will be verified using a SHA-256 cryptographic hash. The models will evaluate each patient record independently in zero-shot isolation, with no cross-contamination or conversational history retention between runs.
REFERENCE STANDARD CONSENSUS PROTOCOL The reference standard consists of a structured consensus HEART score established by three independent emergency medicine physicians (each possessing >=3 years of clinical experience and specific training on HEART score criteria). The physicians will review the anonymized clinical charts while remaining strictly blinded to the LLM outputs and the final 30-day MACE outcomes. For each of the 5 HEART components (scored 0, 1, or 2), a majority vote (2/3 agreement) will determine the final component score. In the event of complete disagreement across all three reviewers on a specific component, a fourth independent adjudicator will resolve the tie.
INDETERMINATE RESULTS MANAGEMENT
In strict compliance with STARD-AI 2025 guidelines, cases with missing or uninterpretable parameters within the free-text clinical notes will be classified into predefined indeterminate tiers:
- Complete Cases: 0 indeterminate components (eligible for primary diagnostic accuracy analysis).
- Partial Indeterminate: Exactly 1 missing component preventing definitive automatic calculation.
- Full Indeterminate: >=2 missing components. The proportion of indeterminate classifications will be quantified for both LLMs and evaluated alongside the routine documentation quality of the charts.
STATISTICAL ANALYSIS AND AGREEMENT WEIGHTING Statistical power and sample size calculation are based on the Hanley-McNeil methodology for the Area Under the ROC Curve (AUC). To achieve an expected AUC of 0.85 with a non-inferiority margin of 0.05, a power of 80%, and a two-sided alpha of 0.05, the primary complete-case analysis requires 600 evaluable patients. Accounting for an anticipated 15% indeterminate rate, a total enrollment target of 690 patients is set. Inter-rater agreement between each LLM and the expert consensus will be computed using quadratic weighted Cohen's Kappa for the ordinal total HEART score (0-10) and linear weighted Kappa for individual components (0-2). Diagnostic performance metrics (sensitivity, specificity, PPV, NPV) will be calculated at prespecified binary (>=4) and trimodal thresholds with 95% Wilson confidence intervals. Pairwise comparison of AUC values between GPT-4o and Claude will be executed using the DeLong test.
DATA ANONYMIZATION AND PRIVACY To ensure full compliance with local personal data protection legislation (KVKK), all free-text emergency department notes will undergo strict de-identification. Patient names, institutional ID numbers, precise dates, and specific demographic identifiers will be stripped entirely before formatting the data payload for API transmission.
PATIENT AND PUBLIC INVOLVEMENT BEYANI Patient and public involvement was not applicable to this study as it involves the analysis of routinely collected clinical data.
- Conditions
- Emergency Medicine, Artificial Intelligence (AI), Artificial Intelligence (AI) in Diagnosis, Chest Pain Rule Out Myocardial Infarction
- Being tested
- GPT-4o HEART Score Calculator, Claude HEART Score Calculator, Three-Expert Consensus HEART Score
- Keywords
- Large Language Model, GPT-4o, Claude Sonnet, Emergency Department, Diagnostic Accuracy, Medical Informatics, Physician vs AI, HEART score
- Medical subject headings
- Emergencies
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
GPT-4o HEART Score CalculatorOther
Also called GPT-4o İndeks Testi
OpenAI GPT-4o (model: gpt-4o-2024-11-20, temperature=0, max_tokens=500, response_format=JSON). Each patient's anonymized anamnesis text, ECG report text, troponin value, and age are submitted via a separate API call with no conversation history. Output: HEART score components (0-2 each), total score (0-10), risk group, and indeterminate status.
Claude HEART Score CalculatorOther
Also called Claude İndeks Testi
Anthropic Claude (model: claude-sonnet-4-6, temperature=0, max_tokens=500, response_format=JSON). Identical system prompt and input format as GPT-4o. Processed independently with no cross-contamination between models. Output: same JSON schema as GPT-4o.
Three-Expert Consensus HEART ScoreOther
Also called Referans Standart
Three emergency medicine physicians (>=3 years experience, HEART-score trained) independently score each anonymized record. Majority vote (2/3) determines component scores; a 4th adjudicator resolves ties. Experts are blinded to LLM scores, each other's scores, and MACE outcomes.
What is being measured
The primary outcome is the main question the study is designed to answer.
Area Under the ROC Curve (AUC) of GPT-4o and Claude HEART Score for 30-Day MACE Prediction
AUC calculated separately for GPT-4o and Claude using the Hanley-McNeil method. MACE is defined as a composite of all-cause death, acute myocardial infarction (Type 1/2/4b), and unplanned revascularization within 30 days. HEART score range is 0-10; a higher score indicates a higher risk of MACE. Analysis will be performed on complete cases only (0 indeterminate components).
Measured: 30 days after index emergency department visit
Secondary outcomes (6)
Sensitivity and Specificity of GPT-4o and Claude HEART Score at Prespecified Thresholds
Diagnostic sensitivity and specificity calculated at two threshold types: (a) total score >=4 (binary high-risk cutoff) and (b) trimodal cutoffs (0-3 low risk, 4-6 intermediate risk, 7-10 high risk). Metrics will be reported with 95% Wilson confidence intervals separately for each LLM.
Measured: 30 days after index emergency department visit
Component-Level and Total-Score Agreement (Cohen's Kappa) Between LLMs and Expert Consensus
Inter-rater agreement will be computed using quadratic weighted Cohen's Kappa for the ordinal total HEART score (range 0-10) and linear weighted Kappa for the individual components (range 0-2). Calculated separately for GPT-4o vs. expert consensus and Claude vs. expert consensus. Values will be interpreted using the Landis \& Koch scale (<0.20 poor, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 good, >0.80 excellent).
Measured: Baseline (At index emergency department visit)
Comparative AUC Difference Between GPT-4o and Claude (DeLong Test)
Statistical comparison of paired ROC curves between GPT-4o and Claude using the DeLong et al. (1988) method. The formal hypothesis is non-inferiority with an expected delta AUC <= 0.05. The correlation coefficient between the paired LLM measurements is estimated as rho >= 0.70.
Measured: 30 days after index emergency department visit
Proportion of Indeterminate Results for GPT-4o and Claude
The proportion of cases classified into predefined missing data tiers: Complete (0 indeterminate components), Partial indeterminate (exactly 1 missing component preventing definitive score calculation), and Full indeterminate (>=2 missing components). Reported separately for each LLM and statistically compared between the two models.
Measured: Baseline (At index emergency department visit)
HEART Parameter Documentation Rate in Routine Turkish Anamnesis Notes
For each of the 5 individual HEART components, the proportion of emergency department free-text anamnesis notes that spontaneously contain sufficient objective clinical information for scoring. Rates will be categorized as: Present and scorable, Partiall
Measured: Baseline (At index emergency department visit)
Subgroup AUC by Age Group and Sex (Algorithmic Bias Assessment)
AUC values for 30-day MACE prediction were calculated separately across demographic strata: age groups (<45, 45-64, >=65 years) and biological sex (male vs. female). This analysis serves as the formal algorithmic bias assessment required by the STARD-AI 2025 guidelines.
Measured: 30 days after the index emergency department visit
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted4 Jun 2026
- Study startJun 2026 (estimated)
- Primary completionMar 2027 (estimated)
- Study completionJun 2027 (estimated)
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Publications (6)Papers the sponsor has linked to this study.
- Other trials for Emergency MedicineStudies of the same condition, open and under way.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Observational
- Observational model
- Cohort
- Time perspective
- Prospective
- Enrolment
- 690 participants (estimated)
Sponsor and investigators
- Lead sponsor
- Marmara University Pendik Training and Research Hospital
- Sponsor type
- University, hospital or other
- Responsible party
- Principal investigator, Emir Ünal, Marmara University Pendik Training and Research Hospital
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT07626060
- Sponsor's protocol ID
- 09.2026.26-0150
- Acronym
- LLM-HEART
- Status last verified
- Jun 2026
- First submitted
- 27 May 2026
- Last updated
- 23 Jun 2026
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- Yes: Anonymized individual participant data (including de-identified baseline demographics, clinical presentation characteristics, index test outputs from GPT-4o and Claude, and the reference standard expert consensus HEART scores) will be made publicly available to support academic transparency and replication. Additionally, the complete deterministic system prompt texts (verified with SHA-256 cryptographic hashes) and the complete statistical analysis code will be included as supplementary material.
Publications
Mahler SA, Riley RF, Hiestand BC, Russell GB, Hoekstra JW, Lefebvre CW, Nicks BA, Cline DM, Askew KL, Elliott SB, Herrington DM, Burke GL, Miller CD. The HEART Pathway randomized trial: identifying emergency department patients with acute chest pain for early discharge. Circ Cardiovasc Qual Outcomes. 2015 Mar;8(2):195-203. doi: 10.1161/CIRCOUTCOMES.114.001384. Epub 2015 Mar 3.
PubMed 25737484Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, Scales N, Tanwani A, Cole-Lewis H, Pfohl S, Payne P, Seneviratne M, Gamble P, Kelly C, Babiker A, Scharli N, Chowdhery A, Mansfield P, Demner-Fushman D, Aguera Y Arcas B, Webster D, Corrado GS, Matias Y, Chou K, Gottweis J, Tomasev N, Liu Y, Rajkomar A, Barral J, Semturs C, Karthikesalingam A, Natarajan V. Large language models encode clinical knowledge. Nature. 2023 Aug;620(7972):172-180. doi: 10.1038/s41586-023-06291-2. Epub 2023 Jul 12.
PubMed 37438534Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, Ghassemi M, Liu X, Reitsma JB, van Smeden M, Boulesteix AL, Camaradou JC, Celi LA, Denaxas S, Denniston AK, Glocker B, Golub RM, Harvey H, Heinze G, Hoffman MM, Kengne AP, Lam E, Lee N, Loder EW, Maier-Hein L, Mateen BA, McCradden MD, Oakden-Rayner L, Ordish J, Parnell R, Rose S, Singh K, Wynants L, Logullo P. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024 Apr 16;385:e078378. doi: 10.1136/bmj-2023-078378.
PubMed 38626948Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, Lijmer JG, Moher D, Rennie D, de Vet HC, Kressel HY, Rifai N, Golub RM, Altman DG, Hooft L, Korevaar DA, Cohen JF; STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015 Oct 28;351:h5527. doi: 10.1136/bmj.h5527.
PubMed 26511519Backus BE, Six AJ, Kelder JC, Bosschaert MA, Mast EG, Mosterd A, Veldkamp RF, Wardeh AJ, Tio R, Braam R, Monnink SH, van Tooren R, Mast TP, van den Akker F, Cramer MJ, Poldervaart JM, Hoes AW, Doevendans PA. A prospective validation of the HEART score for chest pain patients at the emergency department. Int J Cardiol. 2013 Oct 3;168(3):2153-8. doi: 10.1016/j.ijcard.2013.01.255. Epub 2013 Mar 7.
PubMed 23465250Albrecht M. C4-bound imidazolylidenes: from curiosities to high-impact carbene ligands. Chem Commun (Camb). 2008 Aug 21;(31):3601-10. doi: 10.1039/b806924g. Epub 2008 Jul 8.
PubMed 18665276
Study at a glance
From the registry record, last updated by the sponsor on 23 Jun 2026. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Diagnostic Accuracy of Large Language Models (GPT-4o and Claude) in HEART Score Calculation and 30-Day MACE Prediction in Emergency Department Chest Pain Patients: A Prospective Observational Validation Study Against Three-Expert Consensus
- ClinicalTrials.gov ID
- NCT07626060
- Acronym / sponsor ID
- LLM-HEART / 09.2026.26-0150
- Conditions
- Emergency Medicine, Artificial Intelligence (AI), Artificial Intelligence (AI) in Diagnosis, Chest Pain Rule Out Myocardial Infarction
- Keywords
- Large Language Model, GPT-4o, Claude Sonnet, Emergency Department, Diagnostic Accuracy, Medical Informatics, Physician vs AI, HEART score
- Study type
- Observational
- Recruitment status
- Recruiting
- Enrolment
- 690 participants (estimated)
- Lead sponsor
- Marmara University Pendik Training and Research HospitalUniversity, hospital or other
- Study start
- Jun 2026 (estimated)
- Primary completion
- Mar 2027 (estimated)
- Study completion
- Jun 2027 (estimated)
- First posted
- 4 Jun 2026 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
This prospective observational diagnostic accuracy study evaluates whether large language models (LLMs) - GPT-4o (OpenAI, gpt-4o-2024-11-20) and Claude (Anthropic, claude-sonnet-4-6) - can accurately calculate HEART scores from unstructured Turkish clinical notes and predict 30-day major adverse cardiac events (MACE) in emergency department patients presenting with non-traumatic chest pain.
The study will enroll 600 consecutive adult patients. For each patient, the same anonymized data (free-text anamnesis, ECG report text, troponin value, and age) will be independently processed by both LLMs via separate API calls with deterministic settings (temperature=0, JSON format). A three-expert consensus HEART score - derived through blinded independent scoring by three emergency medicine physicians with majority-vote adjudication - serves as the reference standard for agreement analysis. Actual 30-day MACE (all-cause death, AMI Type 1/2/4b, unplanned revascularization) determined via national health database and telephone follow-up serves as the outcome for diagnostic accuracy analysis.
A secondary documentation-quality sub-study will quantify how spontaneously Turkish emergency anamnesis notes capture HEART score parameters.
Detailed description
AI SYSTEM SPECIFICATIONS AND PROMPT PROTOCOL Two distinct large language models (LLMs) will be evaluated as index tests: OpenAI GPT-4o (model string: gpt-4o-2024-11-20) and Anthropic Claude (model string: claude-sonnet-4-6). To ensure reproducibility and eliminate stochastic variation, both models will be accessed via standardized API calls using deterministic parameters (temperature = 0, max_tokens = 500, and strict JSON response format). The exact system prompt layout will be locked prior to initialization, and its integrity will be verified using a SHA-256 cryptographic hash. The models will evaluate each patient record independently in zero-shot isolation, with no cross-contamination or conversational history retention between runs.
REFERENCE STANDARD CONSENSUS PROTOCOL The reference standard consists of a structured consensus HEART score established by three independent emergency medicine physicians (each possessing >=3 years of clinical experience and specific training on HEART score criteria). The physicians will review the anonymized clinical charts while remaining strictly blinded to the LLM outputs and the final 30-day MACE outcomes. For each of the 5 HEART components (scored 0, 1, or 2), a majority vote (2/3 agreement) will determine the final component score. In the event of complete disagreement across all three reviewers on a specific component, a fourth independent adjudicator will resolve the tie.
INDETERMINATE RESULTS MANAGEMENT
In strict compliance with STARD-AI 2025 guidelines, cases with missing or uninterpretable parameters within the free-text clinical notes will be classified into predefined indeterminate tiers:
- Complete Cases: 0 indeterminate components (eligible for primary diagnostic accuracy analysis).
- Partial Indeterminate: Exactly 1 missing component preventing definitive automatic calculation.
- Full Indeterminate: >=2 missing components. The proportion of indeterminate classifications will be quantified for both LLMs and evaluated alongside the routine documentation quality of the charts.
STATISTICAL ANALYSIS AND AGREEMENT WEIGHTING Statistical power and sample size calculation are based on the Hanley-McNeil methodology for the Area Under the ROC Curve (AUC). To achieve an expected AUC of 0.85 with a non-inferiority margin of 0.05, a power of 80%, and a two-sided alpha of 0.05, the primary complete-case analysis requires 600 evaluable patients. Accounting for an anticipated 15% indeterminate rate, a total enrollment target of 690 patients is set. Inter-rater agreement between each LLM and the expert consensus will be computed using quadratic weighted Cohen's Kappa for the ordinal total HEART score (0-10) and linear weighted Kappa for individual components (0-2). Diagnostic performance metrics (sensitivity, specificity, PPV, NPV) will be calculated at prespecified binary (>=4) and trimodal thresholds with 95% Wilson confidence intervals. Pairwise comparison of AUC values between GPT-4o and Claude will be executed using the DeLong test.
DATA ANONYMIZATION AND PRIVACY To ensure full compliance with local personal data protection legislation (KVKK), all free-text emergency department notes will undergo strict de-identification. Patient names, institutional ID numbers, precise dates, and specific demographic identifiers will be stripped entirely before formatting the data payload for API transmission.
PATIENT AND PUBLIC INVOLVEMENT BEYANI Patient and public involvement was not applicable to this study as it involves the analysis of routinely collected clinical data.
Study design
- Observational model
- Cohort
- Time perspective
- Prospective
Who may qualify
- Ages
- 18 years and older
- Sex
- All
- Healthy volunteers
- Not accepted
- Age groups
- Adult, Older adult
- Study population
- The study population consists of consecutive adult patients presenting with a chief complaint of non-traumatic chest pain to the emergency department of Marmara University Pendik Training and Research Hospital, a tertiary care academic medical center in Istanbul, Turkey. This target population comprises real-world emergency medicine admissions that require acute coronary syndrome risk stratification and evaluation with the HEART score. It excludes individuals presenting with traumatic pain etiologies or acute ST-elevation myocardial infarction (STEMI) requiring immediate, time-critical reperfusion pathways.
Key inclusion criteria (4 listed)
- Age >=18 years
- Chief complaint of non-traumatic chest pain at the emergency department
- Written informed consent obtained from the patient or legally authorized representative
- Availability for 30-day follow-up (reachable by telephone and/or actively registered in the e-Nabiz national health database)
Key exclusion criteria (7 listed)
- Traumatic chest pain etiology
- ST-elevation myocardial infarction (STEMI) at presentation requiring immediate reperfusion protocol
- Refusal or subsequent withdrawal of informed consent
- Inability to complete the mandatory 30-day follow-up period
- Patient or representative requests data withdrawal after initial consent
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| GPT-4o HEART Score CalculatorAlso: GPT-4o İndeks TestiOpenAI GPT-4o (model: gpt-4o-2024-11-20, temperature=0, max_tokens=500, response_format=JSON). Each patient's anonymized anamnesis text, ECG report text, troponin value, and age are submitted via a separate API call with no conversation history. Output: HEART score components (0-2 each), total score (0-10), risk group, and indeterminate status. | Other | – |
| Claude HEART Score CalculatorAlso: Claude İndeks TestiAnthropic Claude (model: claude-sonnet-4-6, temperature=0, max_tokens=500, response_format=JSON). Identical system prompt and input format as GPT-4o. Processed independently with no cross-contamination between models. Output: same JSON schema as GPT-4o. | Other | – |
| Three-Expert Consensus HEART ScoreAlso: Referans StandartThree emergency medicine physicians (>=3 years experience, HEART-score trained) independently score each anonymized record. Majority vote (2/3) determines component scores; a 4th adjudicator resolves ties. Experts are blinded to LLM scores, each other's scores, and MACE outcomes. | Other | – |
Primary outcome
Area Under the ROC Curve (AUC) of GPT-4o and Claude HEART Score for 30-Day MACE Prediction
Time frame: 30 days after index emergency department visit
AUC calculated separately for GPT-4o and Claude using the Hanley-McNeil method. MACE is defined as a composite of all-cause death, acute myocardial infarction (Type 1/2/4b), and unplanned revascularization within 30 days. HEART score range is 0-10; a higher score indicates a higher risk of MACE. Analysis will be performed on complete cases only (0 indeterminate components).
6 secondary outcomes
Sensitivity and Specificity of GPT-4o and Claude HEART Score at Prespecified Thresholds
Time frame: 30 days after index emergency department visit
Component-Level and Total-Score Agreement (Cohen's Kappa) Between LLMs and Expert Consensus
Time frame: Baseline (At index emergency department visit)
Comparative AUC Difference Between GPT-4o and Claude (DeLong Test)
Time frame: 30 days after index emergency department visit
Proportion of Indeterminate Results for GPT-4o and Claude
Time frame: Baseline (At index emergency department visit)
HEART Parameter Documentation Rate in Routine Turkish Anamnesis Notes
Time frame: Baseline (At index emergency department visit)
Subgroup AUC by Age Group and Sex (Algorithmic Bias Assessment)
Time frame: 30 days after the index emergency department visit
Results and safety
This study is still under way. Results are not available in the source record yet.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 23 Jun 2026.
- Results publication: PubMed 25737484Mahler SA, Riley RF, Hiestand BC, Russell GB, Hoekstra JW, Lefebvre CW, Nicks BA, Cline DM, Askew KL, Elliott SB, Herrington DM, Burke GL, Miller CD. The HEART Pathway randomized trial: identifying emergency department patients with acute chest pain for early discharge. Circ Cardiovasc Qual Outcomes. 2015 Mar;8(2):195-203. doi: 10.1161/CIRCOUTCOMES.114.001384. Epub 2015 Mar 3.
- Results publication: PubMed 37438534Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, Scales N, Tanwani A, Cole-Lewis H, Pfohl S, Payne P, Seneviratne M, Gamble P, Kelly C, Babiker A, Scharli N, Chowdhery A, Mansfield P, Demner-Fushman D, Aguera Y Arcas B, Webster D, Corrado GS, Matias Y, Chou K, Gottweis J, Tomasev N, Liu Y, Rajkomar A, Barral J, Semturs C, Karthikesalingam A, Natarajan V. Large language models encode clinical knowledge. Nature. 2023 Aug;620(7972):172-180. doi: 10.1038/s41586-023-06291-2. Epub 2023 Jul 12.
- Results publication: PubMed 38626948Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, Ghassemi M, Liu X, Reitsma JB, van Smeden M, Boulesteix AL, Camaradou JC, Celi LA, Denaxas S, Denniston AK, Glocker B, Golub RM, Harvey H, Heinze G, Hoffman MM, Kengne AP, Lam E, Lee N, Loder EW, Maier-Hein L, Mateen BA, McCradden MD, Oakden-Rayner L, Ordish J, Parnell R, Rose S, Singh K, Wynants L, Logullo P. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024 Apr 16;385:e078378. doi: 10.1136/bmj-2023-078378.
- Results publication: PubMed 26511519Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, Lijmer JG, Moher D, Rennie D, de Vet HC, Kressel HY, Rifai N, Golub RM, Altman DG, Hooft L, Korevaar DA, Cohen JF; STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015 Oct 28;351:h5527. doi: 10.1136/bmj.h5527.
- Results publication: PubMed 23465250Backus BE, Six AJ, Kelder JC, Bosschaert MA, Mast EG, Mosterd A, Veldkamp RF, Wardeh AJ, Tio R, Braam R, Monnink SH, van Tooren R, Mast TP, van den Akker F, Cramer MJ, Poldervaart JM, Hoes AW, Doevendans PA. A prospective validation of the HEART score for chest pain patients at the emergency department. Int J Cardiol. 2013 Oct 3;168(3):2153-8. doi: 10.1016/j.ijcard.2013.01.255. Epub 2013 Mar 7.
- Results publication: PubMed 18665276Albrecht M. C4-bound imidazolylidenes: from curiosities to high-impact carbene ligands. Chem Commun (Camb). 2008 Aug 21;(31):3601-10. doi: 10.1039/b806924g. Epub 2008 Jul 8.
- Individual participant dataSharing planned: Yes. Anonymized individual participant data (including de-identified baseline demographics, clinical presentation characteristics, index test outputs from GPT-4o and Claude, and the reference standard expert consensus HEART scores) will be made publicly available to support academic transparency and replication. Additionally, the complete deterministic system prompt texts (verified with SHA-256 cryptographic hashes) and the complete statistical analysis code will be included as supplementary material.
Related trials for Emergency Medicine
Enrolling by invitation
Pediatric Intubation Simulations With Bougie
Difficult Airway Intubation
Recruiting
Restorative Early Sleep Treatment After the Emergency Department
Insomnia, Nightmares Associated With Trauma and Stress
Not yet recruitingPhase 4
Starting HIV Prevention Medicine (PrEP) During an Emergency Department Visit With a Peer Navigator
HIV Prevention, Pre-Exposure Prophylaxis (PrEP)
Not yet recruitingObservational study
High-risk Antithrombotic Drugs at Discharge and Consultation for Hemorrhagic or Thromboembolic Events in the Emergency Department
Acute Disease
RecruitingObservational study
Mechanisms of Myocardial Injury and Ischemia in Patients With Rapid Atrial Fibrillation
Atrial Fibrillation (AF)
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part.
Emre Kudu, associate professor contact
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
An observational study of Emergency Medicine and Artificial Intelligence (AI), run by Marmara University Pendik Training and Research Hospital at 1 site in Turkey (Türkiye).
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 4 inclusion and 7 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: looking for participants now, as of the sponsor's last update on 23 Jun 2026. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in Turkey (Türkiye). Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
This study is still under way. Results are not available in the source record yet.
How can a doctor refer a patient or contact the research team?
Contact Emir Unal, Assistant Professor (+905327766010, emirunal@gmail.com), or a site near the patient. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Area Under the ROC Curve (AUC) of GPT-4o and Claude HEART Score for 30-Day MACE Prediction (time frame: 30 days after index emergency department visit). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT07626060, which also lists 6 linked publications.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 23 Jun 2026.