CommunicationNCT06624605
Enhancing Interdisciplinary Understanding of Ophthalmology Notes Through a Local Large Language Model
A study testing Large Language Model-generated Plain Language Summary of Ophthalmology notes in people with Communication and Artificial Intelligence (AI), run by John J Chen at 1 site in United States.
Trial summary
This prospective, randomized controlled trial evaluated the efficacy of adding Large Language model (LLM)-generated Plain Language Summaries (PLSs) to Standard Ophthalmology Notes (SONs) in enhancing comprehension among non-ophthalmology providers. Read more about this study
- Who is it for?
- Ages 18 years and older
- Conditions the study is for
- Communication, Artificial Intelligence (AI), Artificial Intelligence Technology
- What is being tested?
- Large Language Model-generated Plain Language Summary of Ophthalmology notes
- Could I receive a placebo?
- No, everyone receives an active treatment
- How many people
- 851 participants
- Where
- 1 siteUnited States
- Study dates
- 1 Feb 2024to 31 May 2024
- Phase
- Interventional
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages 18 years and older
- All sexes
- No healthy volunteers
- Adult, Older adult
Who may be able to join
- Evaluation by an Ophthalmologist in the Mayo Clinic Department of Ophthalmology in Rochester, MN in either the inpatient or outpatient setting between Feb 1, 2024 - May 31, 2024.
Who cannot join
- No in-network non-Ophthalmologist care providers in the medical record
- Procedure-only visits containing only a procedure note
Trial locations and contacts
1 site in 1 country.
About this study
Official title: Enhancing Interdisciplinary Understanding of Ophthalmology Notes Through Local Large Language Model
This prospective, randomized controlled trial evaluated the efficacy of adding Large Language model (LLM)-generated Plain Language Summaries (PLSs) to Standard Ophthalmology Notes (SONs) in enhancing comprehension among non-ophthalmology providers. The study utilized surveys to assess non-ophthalmology providers\' comprehension and satisfaction with the notes and ophthalmologists\' evaluation of PLS accuracy, safety, and time burden. An objective semantic and linguistic analysis of the PLSs was also conducted.
- Conditions
- Communication, Artificial Intelligence (AI), Artificial Intelligence Technology, Interdisciplinary Communication
- Being tested
- Large Language Model-generated Plain Language Summary of Ophthalmology notes
- Medical subject headings
- Communication
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
Large Language Model-generated Plain Language Summary of Ophthalmology notesOther
Prospective, randomized Quality Improvement study with real-world implementation of Large Language Model-generated Plain Language Summaries of Ophthalmology notes.
| Study group | Role | What they receive |
|---|---|---|
| Control | No intervention | Participants receive standard ophthalmology notes without any additions. |
| Intervention | Experimental | Participants receive standard ophthalmology notes with an appended LLM-generated plain language summary, reviewed and edited by the ophthalmologist. The plain language summary is clearly marked as generated as part of a Quality Improvement initiative. Receives: Other: Large Language Model-generated Plain Language Summary of Ophthalmology notes |
What is being measured
The primary outcome is the main question the study is designed to answer.
Non-Ophthalmologist Note Comprehension and Satisfaction
Assessed via survey responses evaluating understanding of the patient\'s ophthalmology diagnosis, satisfaction with the note, and overall preference between Standard Ophthalmology Note and Standard Ophthalmology Note + Plain Language Summary.
Degree of understanding was assessed on the following Likert scale:
Not at all Neutral Moderately A great deal
Satisfaction was graded on the following Likert scale:
Not satisfied at all Somewhat unsatisfied Neutral Somewhat satisfied Satisfied
Preference between note types was assessed on the following Likert scale:
Standard Note, a great deal Standard Note, somewhat Neutral Plain Language Summary, somewhat Plain Language Summary, a great deal
Measured: From enrollment to 8 weeks after enrollment
Secondary outcomes (6)
Ophthalmologist Evaluation of Summary Accuracy and Time burden
Assessed via survey responses regarding Plain Language Summary accuracy in reflecting the Standard Ophthalmology Note findings, time spent reviewing/editing, and perceived burden.
Accuracy in reflecting findings was graded on the following Likert scale:
Not at all Neutral A little A great deal
Time spent reviewing/editing was reported using the following options:
\<1 minute
- minute
- minutes
- minutes
- minutes
- or more minutes
Perceived burden was graded on the following Likert scale:
Not at all Neutral A little A great deal
Measured: Assessed at a single time point <24 hr after enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Flesch Reading Ease
Scale: 0 (poor) to 100 (good)
Measured: Assessed at a single time point &lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Flesch-Kincaid Grade Level
Scale: 0 (most difficult to read, highest grade level) to 80 (easiest to read, lowest grade level)
Measured: Assessed at a single time point &lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Semantic Measure of Gobbledygook (SMOG) Index
Scale: 0 (easiest to read) to 11 (most difficult to read)
Measured: Assessed at a single time point &lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
BERTScore Measure of Linguistic Similarity
Scale: -1 (least similar) to +1 (most similar)
Measured: Assessed at a single time point &amp;lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
SBERT Cosine Similarity Measure of Linguistic Similarity
Scale: -1 (least similar) to +1 (most similar)
Measured: Assessed at a single time point &amp;lt;24 hr following enrollment
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted3 Oct 2024
- Study start1 Feb 2024
- Primary completion31 May 2024
- Study completion31 May 2024
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Other trials for CommunicationStudies of the same condition, open and under way.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Health services research
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Model description
- Randomized study assessing the efficacy of Large Language Model-generated Plain Language Summaries of Ophthalmology notes in increasing non-ophthalmologist note understanding. Notes were randomized to intervention (Summary + Original Note) or control (Original note only) arms. Non-ophthalmology providers were sent surveys assessing note comprehension for both control and treatment arm notes, and Ophthalmologists were sent surveys for intervention arm notes regarding Summary accuracy and other attributes. Conducted as a Quality Improvement initiative.
- Masking
- None (open label)
- Enrolment
- 851 participants (actual)
Sponsor and investigators
- Lead sponsor
- John J Chen
- Sponsor type
- University, hospital or other
- Responsible party
- Sponsor-investigator, John J Chen, Mayo Clinic
- Investigators
- John J Chen, MD, PhD, Study director, Mayo Clinic
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT06624605
- Sponsor's protocol ID
- 24-000550
- Status last verified
- Oct 2024
- First submitted
- 30 Sep 2024
- Last updated
- 3 Oct 2024
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- Yes: All collected IPD.
Study at a glance
From the registry record, last updated by the sponsor on 3 Oct 2024. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Enhancing Interdisciplinary Understanding of Ophthalmology Notes Through Local Large Language Model
- ClinicalTrials.gov ID
- NCT06624605
- Acronym / sponsor ID
- 24-000550
- Conditions
- Communication, Artificial Intelligence (AI), Artificial Intelligence Technology, Interdisciplinary Communication
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 851 participants (actual)
- Lead sponsor
- John J ChenUniversity, hospital or other
- Study start
- 1 Feb 2024 (actual)
- Primary completion
- 31 May 2024 (actual)
- Study completion
- 31 May 2024 (actual)
- First posted
- 3 Oct 2024 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
This prospective, randomized controlled trial evaluated the efficacy of adding Large Language model (LLM)-generated Plain Language Summaries (PLSs) to Standard Ophthalmology Notes (SONs) in enhancing comprehension among non-ophthalmology providers. The study utilized surveys to assess non-ophthalmology providers\' comprehension and satisfaction with the notes and ophthalmologists\' evaluation of PLS accuracy, safety, and time burden. An objective semantic and linguistic analysis of the PLSs was also conducted.
Study design
- Primary purpose
- Health services research
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Masking
- None (open label)
- Number of arms
- 2
- Model description
- Randomized study assessing the efficacy of Large Language Model-generated Plain Language Summaries of Ophthalmology notes in increasing non-ophthalmologist note understanding. Notes were randomized to intervention (Summary + Original Note) or control (Original note only) arms. Non-ophthalmology providers were sent surveys assessing note comprehension for both control and treatment arm notes, and Ophthalmologists were sent surveys for intervention arm notes regarding Summary accuracy and other attributes. Conducted as a Quality Improvement initiative.
| Arm | Type | Receives |
|---|---|---|
| Control | No intervention | –Participants receive standard ophthalmology notes without any additions. |
| Intervention | Experimental | Other: Large Language Model-generated Plain Language Summary of Ophthalmology notesParticipants receive standard ophthalmology notes with an appended LLM-generated plain language summary, reviewed and edited by the ophthalmologist. The plain language summary is clearly marked as generated as part of a Quality Improvement initiative. |
Who may qualify
- Ages
- 18 years and older
- Sex
- All
- Healthy volunteers
- Not accepted
- Age groups
- Adult, Older adult
Key inclusion criteria (1 listed)
- Evaluation by an Ophthalmologist in the Mayo Clinic Department of Ophthalmology in Rochester, MN in either the inpatient or outpatient setting between Feb 1, 2024 - May 31, 2024.
Key exclusion criteria (2 listed)
- No in-network non-Ophthalmologist care providers in the medical record
- Procedure-only visits containing only a procedure note
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| Large Language Model-generated Plain Language Summary of Ophthalmology notesProspective, randomized Quality Improvement study with real-world implementation of Large Language Model-generated Plain Language Summaries of Ophthalmology notes. | Other | Intervention |
Primary outcome
Non-Ophthalmologist Note Comprehension and Satisfaction
Time frame: From enrollment to 8 weeks after enrollment
Assessed via survey responses evaluating understanding of the patient\&amp;amp;amp;#39;s ophthalmology diagnosis, satisfaction with the note, and overall preference between Standard Ophthalmology Note and Standard Ophthalmology Note + Plain Language Summary. Degree of understanding was assessed on the following Likert scale: Not at all Neutral Moderately A great deal Satisfaction was graded on the following Likert scale: Not satisfied at all Somewhat unsatisfied Neutral Somewhat satisfied Satisfied Preference between note types was assessed on the following Likert scale: Standard Note, a great deal Standard Note, somewhat Neutral Plain Language Summary, somewhat Plain Language Summary, a great deal
6 secondary outcomes
Ophthalmologist Evaluation of Summary Accuracy and Time burden
Time frame: Assessed at a single time point <24 hr after enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Time frame: Assessed at a single time point &lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Time frame: Assessed at a single time point &lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Time frame: Assessed at a single time point &lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Time frame: Assessed at a single time point &amp;lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Time frame: Assessed at a single time point &amp;lt;24 hr following enrollment
Results and safety
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 3 Oct 2024.
- Individual participant dataSharing planned: Yes. All collected IPD.
Related trials for Communication
RecruitingObservational study
Pediatric Research in the Outpatient Setting CoHort
Factors Associated With a Child's Development of Interaction and Communication Skills and Measures Taken, Reported Stress Level of the Parents of a Newborn and Measures Taken
Not yet recruitingObservational study
Detecting and Predicting Post-stroke Cognitive Impairment Using the Oxford Cognitive Screen
Stroke, Ischemic Stroke
Not yet recruitingObservational study
Validation of the Healthcare Relationship Assessment Scale (ERA) in Health Sciences Students.
Clinical Competence, Health Communication
Recruiting
FOCUS Pre-Post Study
Surgery
Recruiting
The Periviable GOALS Decision Support Tool
Pregnancy Preterm, Premature Birth
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
John J Chen, MD, PhD study director
Mayo Clinic
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing Large Language Model-generated Plain Language Summary of Ophthalmology notes in people with Communication and Artificial Intelligence (AI), run by John J Chen at 1 site in United States.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 1 inclusion and 2 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 3 Oct 2024. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in United States. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Non-Ophthalmologist Note Comprehension and Satisfaction (time frame: From enrollment to 8 weeks after enrollment). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 3 Oct 2024.