Emergency MedicineLLM-ED-DX-TRNCT07632859
Diagnostic Accuracy of Two Large Language Models in Turkish Emergency Department Anamnesis Notes
An observational study of Emergency Medicine and Diagnostic Errors, run by Marmara University Pendik Training and Research Hospital at 1 site in Turkey (Türkiye).
Trial summary
This retrospective diagnostic accuracy study evaluates two large language models - GPT-4.1 (gpt-4.1-2025-04-14; OpenAI) and Claude Sonnet 4.6 (claude-sonnet-4-6; Anthropic) - as retrospective coding-quality instruments applied to anonymized Turkish-language emergency department anamnesis notes. Read more about this study
- Who is it for?
- Ages 18 years and older
- Conditions the study is for
- Emergency Medicine, Diagnostic Errors, Artificial Intelligence (AI) in Diagnosis
- How many people
- 600 participants
- Where
- 1 siteTurkey (Türkiye)
- Study dates
- 1 May 2026to 7 Aug 2026
- Phase
- Observational
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages 18 years and older
- All sexes
- No healthy volunteers
- Adult, Older adult
Study population: The study population comprises consecutive adult patients (aged 18 years and older) evaluated in the ambulatory (green/yellow triage) area of the emergency department of a tertiary care training and research hospital, and whose encounters were documented in the hospital information system (HBYS). Patients triaged to the high-acuity resuscitation area (Emergency Severity Index level 1) were excluded a priori; no Emergency Severity Index level 1 or level 2 presentation occurred within the sampling window, so resuscitation-area presentations are absent from the study population altogether. The findings do not extend to high-acuity emergency care.
Who may be able to join
- Adult patients (aged 18 years and older) presenting to the emergency department, evaluated in the ambulatory (green/yellow triage) area.
- A free-text electronic anamnesis note entered at presentation in the hospital information system (HBYS). No minimum note length and no "sufficient information for diagnosis" requirement was applied, because such a criterion preferentially retains more readily classifiable cases; note length was treated as a covariate rather than as an eligibility threshold. A note was excluded only if all three of the following were absent: any symptom statement, any duration or onset information, and a non-empty anamnesis field.
- An ICD-10 code entered by the treating emergency physician at case closure. Cases in which this entry was absent or did not form a valid ICD-10 code were retained in the analysis set and counted in the denominator of the closure-code analyses.
Who cannot join
- Notes lacking all three of the following: any symptom statement, any duration or onset information, and a non-empty anamnesis field.
- Pediatric cases (age under 18 years).
- Patients critically ill and triaged to high-acuity resuscitation areas (Emergency Severity Index [ESI] level 1).
- Clinical notes containing residual identifying information that cannot be fully de-identified, preventing compliance with data privacy regulations.
- Non-independent clinical notes consisting solely of a brief cross-reference to a prior hospital visit without a new history entry.
Trial locations and contacts
1 site in 1 country.
Turkey (Türkiye)1 site
Marmara University Pendik Training and Research Hospital
Istanbul, Istanbul, 34899
About this study
Official title: Diagnostic Accuracy of Two Large Language Models Against a Blinded Specialist Consensus Standard in Turkish Emergency Department Notes: A Retrospective Study of 600 Cases
This retrospective diagnostic accuracy study evaluates two large language models - GPT-4.1 (gpt-4.1-2025-04-14; OpenAI) and Claude Sonnet 4.6 (claude-sonnet-4-6; Anthropic) - as retrospective coding-quality instruments applied to anonymized Turkish-language emergency department anamnesis notes.
The reference standard is the majority consensus of three board-certified emergency medicine specialists who independently coded each note in ICD-10, blinded to one another, to the code entered by the treating physician at case closure, and to the subsequent clinical course. Cases without chapter-level majority agreement are excluded without replacement.
Read more
Both models are queried once per note with a single locked prompt at temperature 0 in stateless application programming interface calls, with no retrieval augmentation, no external tools and no extended-reasoning mode. The primary outcome is the proportion of cases in which each model's rank-1 diagnosis matches the reference standard at ICD-10 chapter level, reported with a Wilson 95% confidence interval. Registered secondary outcome measures are chapter-level Cohen's kappa between each model's rank-1 diagnosis and the reference standard; top-3 chapter accuracy for each model; and chapter-level concordance between the closure ICD-10 code and the reference standard. Additional prespecified analyses set out in the statistical analysis plan (paired between-model difference, three-character accuracy, note-length association, confidence calibration and model-to-model agreement) are reported in the primary publication.
The ICD-10 code entered at case closure is characterised against the same reference standard as a description of current documentation practice; it is not a comparator, and no test of superiority or inferiority against model output is performed. The analysis plan was finalised and frozen before any accuracy computation. Reporting follows STARD-AI 2025.
Read the detailed description
STUDY DESIGN: Retrospective diagnostic accuracy study, STARD-AI 2025 reporting, single centre, cohort design.
AI INDEX TESTS: (1) GPT-4.1 (model version gpt-4.1-2025-04-14; OpenAI API). (2) Claude Sonnet 4.6 (model version claude-sonnet-4-6; Anthropic API). Both accessed via the providers' developer application programming interfaces from Python. Temperature = 0. Zero-shot direct prompting with a single locked prompt version; stateless single-turn sessions with no cross-case context, no retrieval augmentation, no external tools and no extended-reasoning mode. No task-specific fine-tuning or additional training was applied; the models were used as released.
MODEL INTERPRETABILITY: Interpretability analyses such as SHAP, Grad-CAM or layer-attribution visualisation are not applicable to this study. Because GPT-4.1 and Claude Sonnet 4.6 are accessed as black-box models through proprietary, closed-source commercial interfaces, internal weights, gradients and attention structures are inaccessible for post-hoc interpretability computation.
REFERENCE STANDARD: Three board-certified emergency medicine specialists independently assess each anonymized note, blinded to one another, to the code entered by the treating physician, and to the subsequent clinical course. The primary diagnosis assigned by at least two of three assessors, reduced to ICD-10 chapter level, constitutes the reference standard. Cases in which all three assessors assign different chapters are excluded without replacement. No joint calibration session was held and no adjudication round was performed; each assessor coded once, according to their own clinical judgement.
DATA PRIVACY: All anamnesis notes are de-identified before processing; direct patient identifiers are removed and no patient name is present in any note at any stage. Each case carries a study-specific sequential number that is not a hospital record number, and no file linking study numbers to patient identities was created or retained. Note text is transmitted to commercial application programming interfaces operated by providers established outside Turkiye; all queries are issued through the providers' developer interfaces in stateless single-turn calls, and no patient identifier is present in any submitted text. De-identified notes are stored in an encrypted, access-restricted database. Conducted in accordance with Turkish Personal Data Protection Law no. 6698.
REPRODUCIBILITY OF THE INDEX TEST: The statistical analysis plan specified a test-retest assessment of within-model reproducibility. A random subset of 20 cases was drawn from the analysis set with a fixed seed recorded before the re-run, and both models were re-queried on those notes on 7 August 2026, after an interval of 3 days and 18 hours from the primary run (protocol minimum 48 hours), using the same prompt content, the same model identifiers and the same sampling parameters; the single-query-per-note statement above refers to the primary run. The prespecified measure is the proportion of cases in which the rank-1 code is identical between runs, at three-character and at ICD-10 chapter level. Two conditions differed from the primary run and are recorded in the deviation log: the byte-exact prompt file used in the primary run could not be recovered, only its SHA-256 digest having been retained, so the re-run used a prompt of identical content but unverified byte identity; and the structured-output mechanism for GPT-4.1 was JSON schema mode at re-run rather than the JSON object mode used originally, which the API rejected. The analysis is therefore reported as consistency of re-execution rather than strict prompt-identical reproducibility. This re-run is the last date of data collection and determines the study completion date.
STUDY DATES: The Actual Study Start Date (1 May 2026) denotes the beginning of the retrospective encounter window from which archived notes were drawn, not the start of data collection. Ethics approval (Clinical Research Ethics Committee of Marmara University, protocol 09.2026.26-0514) was granted on 14 May 2026. Because the study is retrospective, every note analysed was already present in the hospital record system when it was retrieved; no data were generated prospectively and no patient was enrolled.
STUDY FLOW: 630 consecutive eligible notes were screened and coded by all three assessors. Ten notes (cases 621-630) fell beyond the ethics-approved ceiling of 600 analysable cases and were excluded before analysis, leaving an assessment window of 620 on which inter-assessor agreement is reported. Within that window 20 notes had no chapter-level majority among the three assessors and were excluded without replacement, giving a primary analysis set of 600. A further 4 notes had no majority three-character code, giving 596 for the secondary three-character analysis.
STATISTICAL ANALYSIS: Analyses are performed in Python 3.11 (pandas, statsmodels, scipy) following a statistical analysis plan finalised and frozen before any accuracy computation; selected estimates are independently recomputed in jamovi by a second investigator using a prespecified verification checklist.
PATIENT AND PUBLIC INVOLVEMENT: Not applicable. This retrospective study uses existing anonymized records; there was no patient or public involvement in design or conduct.
DATA SHARING: De-identified data are available from the principal investigator on reasonable request, subject to institutional approval. Eight supplementary files are provided with the primary publication: the statistical analysis plan with its deviation log; the full prompt text with its recorded SHA-256 digest; the data-preparation and analysis code; the data dictionary; the completed STARD-AI 2025 reporting checklist; the jamovi verification checklist used for independent recomputation of selected estimates; the technical specification of the two index tests; and the full chapter-level confusion matrices for both models.
- Keywords
- Large Language Model; GPT-4o; Claude 4.6 Sonnet; ICD-10; Clinical Coding; Turkish; Emergency Department; Diagnostic Accuracy; STARD; STARD-AI
- Medical subject headings
- Emergencies
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
| Study group | Role | What they receive |
|---|---|---|
| Emergency Department Patient Cohort | Consecutive adult patients (aged 18 years and older) evaluated in the ambulatory (green/yellow triage) area of the emergency department, who had a free-text electronic anamnesis note recorded at presentation and an ICD-10 code entered by the treating physician at case closure. No note-completeness or minimum-length requirement was applied. The closure code is characterised against the reference standard as a description of current documentation practice; it is not a comparator, and no test of superiority or inferiority against model output is performed. |
What is being measured
The primary outcome is the main question the study is designed to answer.
Diagnostic Accuracy of GPT-4.1 for ICD-10 Chapter-Level Diagnosis
Proportion of cases in which the GPT-4.1 primary (rank 1) diagnosis matches the 3-specialist majority-vote reference standard at the ICD-10 chapter level (22 categories). Range: 0 to 1.00.
Measured: At the single index-test run on 3 August 2026
Diagnostic Accuracy of Claude Sonnet 4.6 for ICD-10 Chapter-Level Diagnosis
Proportion of cases in which the Claude Sonnet 4.6 primary (rank 1) diagnosis matches the 3-specialist majority-vote reference standard at the ICD-10 chapter level (22 categories). Range: 0 to 1.00.
Measured: At the single index-test run on 3 August 2026
Secondary outcomes (5)
Cohen's Kappa Between GPT-4.1 Primary Diagnosis and the Reference Standard
Kappa coefficient measuring agreement between the GPT-4.1 rank-1 ICD-10 chapter and the 3-specialist reference standard. Interpreted per Landis \& Koch (1977): <=0.20 slight; 0.21-0.40 fair; 0.41-0.60 moderate; 0.61-0.80 substantial; >0.80 almost perfect. Range: -1.00 to 1.00.
Measured: At the single index-test run on 3 August 2026
Cohen's Kappa Between Claude Sonnet 4.6 Primary Diagnosis and the Reference Standard
Kappa coefficient measuring agreement between the Claude Sonnet 4.6 rank-1 ICD-10 chapter and the 3-specialist reference standard. Interpreted per Landis \& Koch (1977): <=0.20 slight; 0.21-0.40 fair; 0.41-0.60 moderate; 0.61-0.80 substantial; >0.80 almost perfect. Range: -1.00 to 1.00.
Measured: At the single index-test run on 3 August 2026
Top-3 Diagnostic Accuracy of GPT-4.1
Proportion of cases in which the ICD-10 chapter of the reference standard diagnosis appears anywhere within the ranked list of three differential diagnoses returned by GPT-4.1. Range: 0 to 1.00. Cases in which no valid closure code was entered (5 of 600) are retained in the denominator; the figure restricted to resolvable entries is reported alongside.
Measured: At the single index-test run on 3 August 2026
Top-3 Diagnostic Accuracy of Claude Sonnet 4.6
Proportion of cases in which the ICD-10 chapter of the reference standard diagnosis appears anywhere within the ranked list of three differential diagnoses returned by Claude Sonnet 4.6. Range: 0 to 1.00.
Measured: At the single index-test run on 3 August 2026
Chapter-Level Concordance Between the Closure ICD-10 Code and the Reference Standard
Proportion of cases in which the ICD-10 code entered by the treating emergency physician at case closure matches the 3-specialist reference standard at the chapter level. This is reported as a descriptive benchmark of routine coding practice and is not a comparator: the closure code was entered after investigation, whereas the reference standard was constructed from the presentation note alone, to which the assessors were restricted. Range: 0 to 1.00.
Measured: At the original clinical encounter (retrospective data spanning 1 May to 3 August 2026)
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted8 Jun 2026
- Study start1 May 2026
- Primary completion3 Aug 2026
- Study completion7 Aug 2026
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Publications (10)Papers the sponsor has linked to this study.
- Other trials for Emergency MedicineStudies of the same condition, open and under way.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Observational
- Observational model
- Cohort
- Time perspective
- Retrospective
- Enrolment
- 600 participants (actual)
Sponsor and investigators
- Lead sponsor
- Marmara University Pendik Training and Research Hospital
- Sponsor type
- University, hospital or other
- Responsible party
- Principal investigator, Emir Ünal, Marmara University Pendik Training and Research Hospital
- Investigators
- Emir Ünal, Principal investigator, Marmara University
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT07632859
- Sponsor's protocol ID
- 09.2026.26-0514
- Acronym
- LLM-ED-DX-TR
- Status last verified
- Aug 2026
- First submitted
- 3 Jun 2026
- Last updated
- 14 Aug 2026
- Results posted
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
Publications
Sounderajah V, Guni A, Liu X, Collins GS, Karthikesalingam A, Markar SR, Golub RM, Denniston AK, Shetty S, Moher D, Bossuyt PM, Darzi A, Ashrafian H; STARD-AI Steering Committee. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nat Med. 2025 Oct;31(10):3283-3289. doi: 10.1038/s41591-025-03953-8. Epub 2025 Sep 15.
PubMed 40954311Newman-Toker DE, Peterson SM, Badihian S, Hassoon A, Nassery N, Parizadeh D, Wilson LM, Jia Y, Omron R, Tharmarajah S, Guerin L, Bastani PB, Fracica EA, Kotwal S, Robinson KA. Diagnostic Errors in the Emergency Department: A Systematic Review [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2022 Dec. Report No.: 22(23)-EHC043. Available from http://www.ncbi.nlm.nih.gov/books/NBK588118/
PubMed 36574484Wei J et al. Chain-of-thought prompting elicits reasoning in LLMs. NeurIPS. 2022;35:24824-24837.
Niset A, Melot I, Pireau M, Englebert A, Scius N, Flament J, El Hadwe S, Al Barajraji M, Thonon H, Barrit S. Grounded large language models for diagnostic prediction in real-world emergency department settings. JAMIA Open. 2025 Oct 21;8(5):ooaf119. doi: 10.1093/jamiaopen/ooaf119. eCollection 2025 Oct.
PubMed 41127256Williams CYK, Miao BY, Kornblith AE, Butte AJ. Evaluating the use of large language models to provide clinical recommendations in the Emergency Department. Nat Commun. 2024 Oct 8;15(1):8236. doi: 10.1038/s41467-024-52415-1.
PubMed 39379357Hoppe JM, Auer MK, Struven A, Massberg S, Stremmel C. ChatGPT With GPT-4 Outperforms Emergency Department Physicians in Diagnostic Accuracy: Retrospective Analysis. J Med Internet Res. 2024 Jul 8;26:e56110. doi: 10.2196/56110.
PubMed 38976865Kanjee Z, Crowe B, Rodman A. Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge. JAMA. 2023 Jul 3;330(1):78-80. doi: 10.1001/jama.2023.8288.
PubMed 37318797Takita H, Kabata D, Walston SL, Tatekawa H, Saito K, Tsujimoto Y, Miki Y, Ueda D. A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians. NPJ Digit Med. 2025 Mar 22;8(1):175. doi: 10.1038/s41746-025-01543-z.
PubMed 40121370Shan G, Chen X, Wang C, Liu L, Gu Y, Jiang H, Shi T. Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis. JMIR Med Inform. 2025 Apr 25;13:e64963. doi: 10.2196/64963.
PubMed 40279517Taylor RA, Sangal RB, Smith ME, Haimovich AD, Rodman A, Iscoe MS, Pavuluri SK, Rose C, Janke AT, Wright DS, Socrates V, Declan A. Leveraging artificial intelligence to reduce diagnostic errors in emergency medicine: Challenges, opportunities, and future directions. Acad Emerg Med. 2025 Mar;32(3):327-339. doi: 10.1111/acem.15066. Epub 2024 Dec 15.
PubMed 39676165
Study at a glance
From the registry record, last updated by the sponsor on 14 Aug 2026. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Diagnostic Accuracy of Two Large Language Models Against a Blinded Specialist Consensus Standard in Turkish Emergency Department Notes: A Retrospective Study of 600 Cases
- ClinicalTrials.gov ID
- NCT07632859
- Acronym / sponsor ID
- LLM-ED-DX-TR / 09.2026.26-0514
- Keywords
- Large Language Model; GPT-4o; Claude 4.6 Sonnet; ICD-10; Clinical Coding; Turkish; Emergency Department; Diagnostic Accuracy; STARD; STARD-AI
- Study type
- Observational
- Recruitment status
- Completed
- Enrolment
- 600 participants (actual)
- Lead sponsor
- Marmara University Pendik Training and Research HospitalUniversity, hospital or other
- Study start
- 1 May 2026 (actual)
- Primary completion
- 3 Aug 2026 (actual)
- Study completion
- 7 Aug 2026 (actual)
- First posted
- 8 Jun 2026 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
This retrospective diagnostic accuracy study evaluates two large language models - GPT-4.1 (gpt-4.1-2025-04-14; OpenAI) and Claude Sonnet 4.6 (claude-sonnet-4-6; Anthropic) - as retrospective coding-quality instruments applied to anonymized Turkish-language emergency department anamnesis notes.
The reference standard is the majority consensus of three board-certified emergency medicine specialists who independently coded each note in ICD-10, blinded to one another, to the code entered by the treating physician at case closure, and to the subsequent clinical course. Cases without chapter-level majority agreement are excluded without replacement.
Both models are queried once per note with a single locked prompt at temperature 0 in stateless application programming interface calls, with no retrieval augmentation, no external tools and no extended-reasoning mode. The primary outcome is the proportion of cases in which each model's rank-1 diagnosis matches the reference standard at ICD-10 chapter level, reported with a Wilson 95% confidence interval. Registered secondary outcome measures are chapter-level Cohen's kappa between each model's rank-1 diagnosis and the reference standard; top-3 chapter accuracy for each model; and chapter-level concordance between the closure ICD-10 code and the reference standard. Additional prespecified analyses set out in the statistical analysis plan (paired between-model difference, three-character accuracy, note-length association, confidence calibration and model-to-model agreement) are reported in the primary publication.
The ICD-10 code entered at case closure is characterised against the same reference standard as a description of current documentation practice; it is not a comparator, and no test of superiority or inferiority against model output is performed. The analysis plan was finalised and frozen before any accuracy computation. Reporting follows STARD-AI 2025.
Detailed description
STUDY DESIGN: Retrospective diagnostic accuracy study, STARD-AI 2025 reporting, single centre, cohort design.
AI INDEX TESTS: (1) GPT-4.1 (model version gpt-4.1-2025-04-14; OpenAI API). (2) Claude Sonnet 4.6 (model version claude-sonnet-4-6; Anthropic API). Both accessed via the providers' developer application programming interfaces from Python. Temperature = 0. Zero-shot direct prompting with a single locked prompt version; stateless single-turn sessions with no cross-case context, no retrieval augmentation, no external tools and no extended-reasoning mode. No task-specific fine-tuning or additional training was applied; the models were used as released.
MODEL INTERPRETABILITY: Interpretability analyses such as SHAP, Grad-CAM or layer-attribution visualisation are not applicable to this study. Because GPT-4.1 and Claude Sonnet 4.6 are accessed as black-box models through proprietary, closed-source commercial interfaces, internal weights, gradients and attention structures are inaccessible for post-hoc interpretability computation.
REFERENCE STANDARD: Three board-certified emergency medicine specialists independently assess each anonymized note, blinded to one another, to the code entered by the treating physician, and to the subsequent clinical course. The primary diagnosis assigned by at least two of three assessors, reduced to ICD-10 chapter level, constitutes the reference standard. Cases in which all three assessors assign different chapters are excluded without replacement. No joint calibration session was held and no adjudication round was performed; each assessor coded once, according to their own clinical judgement.
DATA PRIVACY: All anamnesis notes are de-identified before processing; direct patient identifiers are removed and no patient name is present in any note at any stage. Each case carries a study-specific sequential number that is not a hospital record number, and no file linking study numbers to patient identities was created or retained. Note text is transmitted to commercial application programming interfaces operated by providers established outside Turkiye; all queries are issued through the providers' developer interfaces in stateless single-turn calls, and no patient identifier is present in any submitted text. De-identified notes are stored in an encrypted, access-restricted database. Conducted in accordance with Turkish Personal Data Protection Law no. 6698.
REPRODUCIBILITY OF THE INDEX TEST: The statistical analysis plan specified a test-retest assessment of within-model reproducibility. A random subset of 20 cases was drawn from the analysis set with a fixed seed recorded before the re-run, and both models were re-queried on those notes on 7 August 2026, after an interval of 3 days and 18 hours from the primary run (protocol minimum 48 hours), using the same prompt content, the same model identifiers and the same sampling parameters; the single-query-per-note statement above refers to the primary run. The prespecified measure is the proportion of cases in which the rank-1 code is identical between runs, at three-character and at ICD-10 chapter level. Two conditions differed from the primary run and are recorded in the deviation log: the byte-exact prompt file used in the primary run could not be recovered, only its SHA-256 digest having been retained, so the re-run used a prompt of identical content but unverified byte identity; and the structured-output mechanism for GPT-4.1 was JSON schema mode at re-run rather than the JSON object mode used originally, which the API rejected. The analysis is therefore reported as consistency of re-execution rather than strict prompt-identical reproducibility. This re-run is the last date of data collection and determines the study completion date.
STUDY DATES: The Actual Study Start Date (1 May 2026) denotes the beginning of the retrospective encounter window from which archived notes were drawn, not the start of data collection. Ethics approval (Clinical Research Ethics Committee of Marmara University, protocol 09.2026.26-0514) was granted on 14 May 2026. Because the study is retrospective, every note analysed was already present in the hospital record system when it was retrieved; no data were generated prospectively and no patient was enrolled.
STUDY FLOW: 630 consecutive eligible notes were screened and coded by all three assessors. Ten notes (cases 621-630) fell beyond the ethics-approved ceiling of 600 analysable cases and were excluded before analysis, leaving an assessment window of 620 on which inter-assessor agreement is reported. Within that window 20 notes had no chapter-level majority among the three assessors and were excluded without replacement, giving a primary analysis set of 600. A further 4 notes had no majority three-character code, giving 596 for the secondary three-character analysis.
STATISTICAL ANALYSIS: Analyses are performed in Python 3.11 (pandas, statsmodels, scipy) following a statistical analysis plan finalised and frozen before any accuracy computation; selected estimates are independently recomputed in jamovi by a second investigator using a prespecified verification checklist.
PATIENT AND PUBLIC INVOLVEMENT: Not applicable. This retrospective study uses existing anonymized records; there was no patient or public involvement in design or conduct.
DATA SHARING: De-identified data are available from the principal investigator on reasonable request, subject to institutional approval. Eight supplementary files are provided with the primary publication: the statistical analysis plan with its deviation log; the full prompt text with its recorded SHA-256 digest; the data-preparation and analysis code; the data dictionary; the completed STARD-AI 2025 reporting checklist; the jamovi verification checklist used for independent recomputation of selected estimates; the technical specification of the two index tests; and the full chapter-level confusion matrices for both models.
Study design
- Observational model
- Cohort
- Time perspective
- Retrospective
- Number of arms
- 1
| Arm | Type | Receives |
|---|---|---|
| Emergency Department Patient Cohort | – | –Consecutive adult patients (aged 18 years and older) evaluated in the ambulatory (green/yellow triage) area of the emergency department, who had a free-text electronic anamnesis note recorded at presentation and an ICD-10 code entered by the treating physician at case closure. No note-completeness or minimum-length requirement was applied. The closure code is characterised against the reference standard as a description of current documentation practice; it is not a comparator, and no test of superiority or inferiority against model output is performed. |
Who may qualify
- Ages
- 18 years and older
- Sex
- All
- Healthy volunteers
- Not accepted
- Age groups
- Adult, Older adult
- Study population
- The study population comprises consecutive adult patients (aged 18 years and older) evaluated in the ambulatory (green/yellow triage) area of the emergency department of a tertiary care training and research hospital, and whose encounters were documented in the hospital information system (HBYS). Patients triaged to the high-acuity resuscitation area (Emergency Severity Index level 1) were excluded a priori; no Emergency Severity Index level 1 or level 2 presentation occurred within the sampling window, so resuscitation-area presentations are absent from the study population altogether. The findings do not extend to high-acuity emergency care.
Key inclusion criteria (3 listed)
- Adult patients (aged 18 years and older) presenting to the emergency department, evaluated in the ambulatory (green/yellow triage) area.
- A free-text electronic anamnesis note entered at presentation in the hospital information system (HBYS). No minimum note length and no "sufficient information for diagnosis" requirement was applied, because such a criterion preferentially retains more readily classifiable cases; note length was treated as a covariate rather than as an eligibility threshold. A note was excluded only if all three of the following were absent: any symptom statement, any duration or onset information, and a non-empty anamnesis field.
- An ICD-10 code entered by the treating emergency physician at case closure. Cases in which this entry was absent or did not form a valid ICD-10 code were retained in the analysis set and counted in the denominator of the closure-code analyses.
Key exclusion criteria (5 listed)
- Notes lacking all three of the following: any symptom statement, any duration or onset information, and a non-empty anamnesis field.
- Pediatric cases (age under 18 years).
- Patients critically ill and triaged to high-acuity resuscitation areas (Emergency Severity Index [ESI] level 1).
- Clinical notes containing residual identifying information that cannot be fully de-identified, preventing compliance with data privacy regulations.
- Non-independent clinical notes consisting solely of a brief cross-reference to a prior hospital visit without a new history entry.
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Primary outcome
Diagnostic Accuracy of GPT-4.1 for ICD-10 Chapter-Level Diagnosis
Time frame: At the single index-test run on 3 August 2026
Proportion of cases in which the GPT-4.1 primary (rank 1) diagnosis matches the 3-specialist majority-vote reference standard at the ICD-10 chapter level (22 categories). Range: 0 to 1.00.
Diagnostic Accuracy of Claude Sonnet 4.6 for ICD-10 Chapter-Level Diagnosis
Time frame: At the single index-test run on 3 August 2026
Proportion of cases in which the Claude Sonnet 4.6 primary (rank 1) diagnosis matches the 3-specialist majority-vote reference standard at the ICD-10 chapter level (22 categories). Range: 0 to 1.00.
5 secondary outcomes
Cohen's Kappa Between GPT-4.1 Primary Diagnosis and the Reference Standard
Time frame: At the single index-test run on 3 August 2026
Cohen's Kappa Between Claude Sonnet 4.6 Primary Diagnosis and the Reference Standard
Time frame: At the single index-test run on 3 August 2026
Top-3 Diagnostic Accuracy of GPT-4.1
Time frame: At the single index-test run on 3 August 2026
Top-3 Diagnostic Accuracy of Claude Sonnet 4.6
Time frame: At the single index-test run on 3 August 2026
Chapter-Level Concordance Between the Closure ICD-10 Code and the Reference Standard
Time frame: At the original clinical encounter (retrospective data spanning 1 May to 3 August 2026)
Results and safety
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 14 Aug 2026.
- Background: PubMed 40954311Sounderajah V, Guni A, Liu X, Collins GS, Karthikesalingam A, Markar SR, Golub RM, Denniston AK, Shetty S, Moher D, Bossuyt PM, Darzi A, Ashrafian H; STARD-AI Steering Committee. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nat Med. 2025 Oct;31(10):3283-3289. doi: 10.1038/s41591-025-03953-8. Epub 2025 Sep 15.
- Background: PubMed 36574484Newman-Toker DE, Peterson SM, Badihian S, Hassoon A, Nassery N, Parizadeh D, Wilson LM, Jia Y, Omron R, Tharmarajah S, Guerin L, Bastani PB, Fracica EA, Kotwal S, Robinson KA. Diagnostic Errors in the Emergency Department: A Systematic Review [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2022 Dec. Report No.: 22(23)-EHC043. Available from http://www.ncbi.nlm.nih.gov/books/NBK588118/
- BackgroundWei J et al. Chain-of-thought prompting elicits reasoning in LLMs. NeurIPS. 2022;35:24824-24837.
- Background: PubMed 41127256Niset A, Melot I, Pireau M, Englebert A, Scius N, Flament J, El Hadwe S, Al Barajraji M, Thonon H, Barrit S. Grounded large language models for diagnostic prediction in real-world emergency department settings. JAMIA Open. 2025 Oct 21;8(5):ooaf119. doi: 10.1093/jamiaopen/ooaf119. eCollection 2025 Oct.
- Background: PubMed 39379357Williams CYK, Miao BY, Kornblith AE, Butte AJ. Evaluating the use of large language models to provide clinical recommendations in the Emergency Department. Nat Commun. 2024 Oct 8;15(1):8236. doi: 10.1038/s41467-024-52415-1.
- Background: PubMed 38976865Hoppe JM, Auer MK, Struven A, Massberg S, Stremmel C. ChatGPT With GPT-4 Outperforms Emergency Department Physicians in Diagnostic Accuracy: Retrospective Analysis. J Med Internet Res. 2024 Jul 8;26:e56110. doi: 10.2196/56110.
- Background: PubMed 37318797Kanjee Z, Crowe B, Rodman A. Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge. JAMA. 2023 Jul 3;330(1):78-80. doi: 10.1001/jama.2023.8288.
- Background: PubMed 40121370Takita H, Kabata D, Walston SL, Tatekawa H, Saito K, Tsujimoto Y, Miki Y, Ueda D. A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians. NPJ Digit Med. 2025 Mar 22;8(1):175. doi: 10.1038/s41746-025-01543-z.
- Background: PubMed 40279517Shan G, Chen X, Wang C, Liu L, Gu Y, Jiang H, Shi T. Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis. JMIR Med Inform. 2025 Apr 25;13:e64963. doi: 10.2196/64963.
- Background: PubMed 39676165Taylor RA, Sangal RB, Smith ME, Haimovich AD, Rodman A, Iscoe MS, Pavuluri SK, Rose C, Janke AT, Wright DS, Socrates V, Declan A. Leveraging artificial intelligence to reduce diagnostic errors in emergency medicine: Challenges, opportunities, and future directions. Acad Emerg Med. 2025 Mar;32(3):327-339. doi: 10.1111/acem.15066. Epub 2024 Dec 15.
Related trials for Emergency Medicine
Enrolling by invitation
Pediatric Intubation Simulations With Bougie
Difficult Airway Intubation
Recruiting
Restorative Early Sleep Treatment After the Emergency Department
Insomnia, Nightmares Associated With Trauma and Stress
Not yet recruitingPhase 4
Starting HIV Prevention Medicine (PrEP) During an Emergency Department Visit With a Peer Navigator
HIV Prevention, Pre-Exposure Prophylaxis (PrEP)
Not yet recruitingObservational study
High-risk Antithrombotic Drugs at Discharge and Consultation for Hemorrhagic or Thromboembolic Events in the Emergency Department
Acute Disease
RecruitingObservational study
Mechanisms of Myocardial Injury and Ischemia in Patients With Rapid Atrial Fibrillation
Atrial Fibrillation (AF)
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
Emir Ünal principal investigator
Marmara University
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
An observational study of Emergency Medicine and Diagnostic Errors, run by Marmara University Pendik Training and Research Hospital at 1 site in Turkey (Türkiye).
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 3 inclusion and 5 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 14 Aug 2026. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in Turkey (Türkiye). Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Diagnostic Accuracy of GPT-4.1 for ICD-10 Chapter-Level Diagnosis (time frame: At the single index-test run on 3 August 2026). The study lists 2 primary outcomes in all. It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT07632859, which also lists 10 linked publications.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 14 Aug 2026.