Clinician-in-the-loopNCT07187050
LLM-Assisted vs Manual Writing for Clinical Documentation: Effects on Time and Quality
A study testing Template-Based LLM Assistant and Manual Writing in people with Clinician-in-the-loop and Clinical Documentation, run by Kyoto University, Graduate School of Medicine at 1 site in Japan.
Trial summary
The goal of this clinical trial is to learn whether an LLM-assisted writing workflow can reduce the time to complete hospital discharge summaries and discharge referrals and maintain or improve document quality compared with writing from scratch by clinicians. Read more about this study
- Who is it for?
- Any age
- Conditions the study is for
- Clinician-in-the-loop, Clinical Documentation, Large Language Model
- What is being tested?
- Template-Based LLM Assistant and Manual Writing
- Could I receive a placebo?
- No, everyone receives an active treatment
- How many people
- 21 participants
- Where
- 1 siteJapan
- Study dates
- 18 Feb 2025to 16 Jul 2025
- Phase
- Interventional
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages Any age
- All sexes
- No healthy volunteers
- Child, Adult, Older adult
Who may be able to join
- Ophthalmologists at Kyoto University Hospital
- Junior residents, senior residents, graduate students, board-certified ophthalmologists
- Physicians who confirm that they do not routinely use CocktailAI for clinical documentation and provide informed consent after receiving an explanation of the study.
Who cannot join
None listed.
Trial locations and contacts
1 site in 1 country.
Japan1 site
Department of Ophthalmology and Visual Sciences Kyoto University Graduate School of Medicine 54 Shogoin, Kawahara, Sakyo
Kyoto, 606-8507
About this study
Official title: Evaluating an LLM-Assisted Workflow for Clinical Documentation: A Pilot Randomized Controlled Trial on Time and Quality
The goal of this clinical trial is to learn whether an LLM-assisted writing workflow can reduce the time to complete hospital discharge summaries and discharge referrals and maintain or improve document quality compared with writing from scratch by clinicians. The study used six simulated patient records (no real patient data).
The main questions it aims to answer are:
Read more
- Does the LLM-assisted writing workflow reduce the time needed to complete each document compared with manual writing?
- Does the LLM-assisted writing workflow improve (or at least maintain) document quality compared with manual writing, as rated by blinded experts?
Researchers will compare LLM-assisted versus manual writing to see if the LLM-assisted approach is faster and has equal or better quality. LLM-only drafts (unedited first drafts) will be evaluated as a separate third group to understand the baseline quality of LLM output without clinician edits.
Participants will create two documents-a discharge summary and a discharge referral-for each of six simulated cases. Those assigned to CocktailAI \& Modification group will use an LLM assistant (called CocktailAI) to generate a first draft for each document and then review and edit it to finalize; those assigned to the control group will write each document from scratch without LLM assistance.
- Being tested
- Template-Based LLM Assistant, Manual Writing
- Keywords
- clinician-in-the-loop, Large Language Model, Clinical Documentation, Electronic Health Record
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
Template-Based LLM AssistantOther
This study uses CocktailAI, a template-based LLM assistant co-developed by the Department of Ophthalmology and Visual Sciences, Kyoto University Graduate School of Medicine, and Fitting Cloud Inc. (Kyoto, Japan). It is designed to extract relevant information from EHRs using LLMs and embed the extracted content into predefined templates. In this trial, the inputs are six simulated patient records (no real patient data). Text generation uses Gemini-2.0-flash-lite. Templates for discharge summaries and discharge referrals are pre-defined by a team member.
Manual WritingOther
The same document templates are provided; however, all LLM instruction prompts are removed in advance. Clinicians manually write the documents, following the template structure, for each of the six simulated cases.
| Study group | Role | What they receive |
|---|---|---|
| CocktailAI & Modification arm | Experimental | Participants log into the study web application using a unique ID and their name. For each simulated patient case, they press "Start" to unlock the case record and initiate timing. After viewing the patient information, clinicians use CocktailAI to generate a first draft of the discharge summary. They then review and edit the draft and submit the final document by pressing "Submit." For the discharge referral of the same case, clinicians follow the same procedure. Copy-and-paste from the case record is permitted. This sequence is repeated for six simulated cases. Receives: Other: Template-Based LLM Assistant |
| Control arm | Active comparator | Participants log into the study web application using a unique ID and their name. For each simulated patient case, they press "Start" to unlock the case record and initiate timing. After viewing the patient information, clinicians write the discharge summary from scratch and submit the document by pressing "Submit." They then write the discharge referral for the same case from scratch and submit it. Copy-and-paste from the case record is permitted. This sequence is repeated for six simulated cases. Receives: Other: Manual Writing |
| CocktailAI arm | Active comparator | The LLM assistant generates drafts of discharge summaries and discharge referrals directly from the simulated patient records, with no clinician review or edits. These drafts are the unedited first drafts produced in the CocktailAI \& Modification arm (captured before any clinician edits) and are included as a separate group. Receives: Other: Template-Based LLM Assistant |
What is being measured
The primary outcome is the main question the study is designed to answer.
Average time spent creating each document
In the CocktailAI \& Modification group and the Control group, the time spent creating documents is measured in seconds. In the CocktailAI group, the time required for document generation is measured in seconds.
Measured: On one study day within 2 weeks after enrollment
Secondary outcomes (1)
Document quality assessment
Blinded to group allocation, ophthalmology experts evaluate the documents using pre-specified criteria defined before study initiation. These criteria are developed based on six domains: Medical Accuracy, Language, Conciseness, Presence of Hallucinations, Validity for Clinical Use, and Possibility of Harm. Most domains are assessed on a three-point scale, whereas Presence of Hallucinations is evaluated dichotomously (present or absent). In addition to these domain-specific ratings, experts provide a subjective overall score on a 10-point scale (with higher scores indicating better quality) and are asked to guess which study group the document belongs to.
Measured: On one study day within 2 weeks after enrollment
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted22 Sep 2025
- Study start18 Feb 2025
- Primary completion14 Mar 2025
- Study completion16 Jul 2025
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Publications (1)Papers the sponsor has linked to this study.
- Other trials for Clinician-in-the-loopStudies of the same condition, open and under way.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Health services research
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Model description
- This is a prospective, randomized, open-label, blinded-endpoint (PROBE) trial.
- Masking
- Single blind
- Who is masked
- Outcomes assessor
- Enrolment
- 21 participants (actual)
Sponsor and investigators
- Lead sponsor
- Kyoto University, Graduate School of Medicine
- Sponsor type
- University, hospital or other
- Collaborators
- Fitting Cloud Inc.
- Responsible party
- Principal investigator, Tomohiro Takayama, Kyoto University, Graduate School of Medicine
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT07187050
- Sponsor's protocol ID
- clinician-in-the-loop_workflow
- Status last verified
- Sep 2025
- First submitted
- 2 Sep 2025
- Last updated
- 22 Sep 2025
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
Publications
Takayama T, Sado K, Suda K, Tamura H, Ueda-Arakawa N, Ishihara K, Ueda Y, Okamoto K, Santos LHO, Oshika T, Kuroda T, Tsujikawa A, Miyake M. Evaluating an LLM-assisted workflow for clinical documentation: A pilot randomized controlled trial on time and quality. PLOS Digit Health. 2026 Oct 26;5(10):e0001799. doi: 10.1371/journal.pdig.0001799. eCollection 2026 Oct.
PubMed 42932273
Study at a glance
From the registry record, last updated by the sponsor on 22 Sep 2025. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Evaluating an LLM-Assisted Workflow for Clinical Documentation: A Pilot Randomized Controlled Trial on Time and Quality
- ClinicalTrials.gov ID
- NCT07187050
- Acronym / sponsor ID
- clinician-in-the-loop_workflow
- Keywords
- clinician-in-the-loop, Large Language Model, Clinical Documentation, Electronic Health Record
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 21 participants (actual)
- Lead sponsor
- Kyoto University, Graduate School of MedicineUniversity, hospital or other
- Collaborators
- Fitting Cloud Inc.
- Study start
- 18 Feb 2025 (actual)
- Primary completion
- 14 Mar 2025 (actual)
- Study completion
- 16 Jul 2025 (actual)
- First posted
- 22 Sep 2025 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
The goal of this clinical trial is to learn whether an LLM-assisted writing workflow can reduce the time to complete hospital discharge summaries and discharge referrals and maintain or improve document quality compared with writing from scratch by clinicians. The study used six simulated patient records (no real patient data).
The main questions it aims to answer are:
- Does the LLM-assisted writing workflow reduce the time needed to complete each document compared with manual writing?
- Does the LLM-assisted writing workflow improve (or at least maintain) document quality compared with manual writing, as rated by blinded experts?
Researchers will compare LLM-assisted versus manual writing to see if the LLM-assisted approach is faster and has equal or better quality. LLM-only drafts (unedited first drafts) will be evaluated as a separate third group to understand the baseline quality of LLM output without clinician edits.
Participants will create two documents-a discharge summary and a discharge referral-for each of six simulated cases. Those assigned to CocktailAI \& Modification group will use an LLM assistant (called CocktailAI) to generate a first draft for each document and then review and edit it to finalize; those assigned to the control group will write each document from scratch without LLM assistance.
Study design
- Primary purpose
- Health services research
- Allocation
- Randomised
- Intervention model
- Parallel groups
- Masking
- Single blind (outcomes assessor)
- Number of arms
- 3
- Model description
- This is a prospective, randomized, open-label, blinded-endpoint (PROBE) trial.
| Arm | Type | Receives |
|---|---|---|
| CocktailAI & Modification arm | Experimental | Other: Template-Based LLM AssistantParticipants log into the study web application using a unique ID and their name. For each simulated patient case, they press "Start" to unlock the case record and initiate timing. After viewing the patient information, clinicians use CocktailAI to generate a first draft of the discharge summary. They then review and edit the draft and submit the final document by pressing "Submit." For the discharge referral of the same case, clinicians follow the same procedure. Copy-and-paste from the case record is permitted. This sequence is repeated for six simulated cases. |
| Control arm | Active comparator | Other: Manual WritingParticipants log into the study web application using a unique ID and their name. For each simulated patient case, they press "Start" to unlock the case record and initiate timing. After viewing the patient information, clinicians write the discharge summary from scratch and submit the document by pressing "Submit." They then write the discharge referral for the same case from scratch and submit it. Copy-and-paste from the case record is permitted. This sequence is repeated for six simulated cases. |
| CocktailAI arm | Active comparator | Other: Template-Based LLM AssistantThe LLM assistant generates drafts of discharge summaries and discharge referrals directly from the simulated patient records, with no clinician review or edits. These drafts are the unedited first drafts produced in the CocktailAI \& Modification arm (captured before any clinician edits) and are included as a separate group. |
Who may qualify
- Sex
- All
- Healthy volunteers
- Not accepted
- Age groups
- Child, Adult, Older adult
Key inclusion criteria (3 listed)
- Ophthalmologists at Kyoto University Hospital
- Junior residents, senior residents, graduate students, board-certified ophthalmologists
- Physicians who confirm that they do not routinely use CocktailAI for clinical documentation and provide informed consent after receiving an explanation of the study.
Key exclusion criteria (0 listed)
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| Template-Based LLM AssistantThis study uses CocktailAI, a template-based LLM assistant co-developed by the Department of Ophthalmology and Visual Sciences, Kyoto University Graduate School of Medicine, and Fitting Cloud Inc. (Kyoto, Japan). It is designed to extract relevant information from EHRs using LLMs and embed the extracted content into predefined templates. In this trial, the inputs are six simulated patient records (no real patient data). Text generation uses Gemini-2.0-flash-lite. Templates for discharge summaries and discharge referrals are pre-defined by a team member. | Other | CocktailAI & Modification arm, CocktailAI arm |
| Manual WritingThe same document templates are provided; however, all LLM instruction prompts are removed in advance. Clinicians manually write the documents, following the template structure, for each of the six simulated cases. | Other | Control arm |
Primary outcome
Average time spent creating each document
Time frame: On one study day within 2 weeks after enrollment
In the CocktailAI \& Modification group and the Control group, the time spent creating documents is measured in seconds. In the CocktailAI group, the time required for document generation is measured in seconds.
1 secondary outcome
Document quality assessment
Time frame: On one study day within 2 weeks after enrollment
Results and safety
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 22 Sep 2025.
- Derived from this study: PubMed 42932273Takayama T, Sado K, Suda K, Tamura H, Ueda-Arakawa N, Ishihara K, Ueda Y, Okamoto K, Santos LHO, Oshika T, Kuroda T, Tsujikawa A, Miyake M. Evaluating an LLM-assisted workflow for clinical documentation: A pilot randomized controlled trial on time and quality. PLOS Digit Health. 2026 Oct 26;5(10):e0001799. doi: 10.1371/journal.pdig.0001799. eCollection 2026 Oct.
Related trials for Clinician-in-the-loop
Active, not recruiting
LLM Psychiatric Intake AI Clinical Performance Study
Mental Disorders
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
The record lists no contact. Use the official record to reach the sponsor.
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing Template-Based LLM Assistant and Manual Writing in people with Clinician-in-the-loop and Clinical Documentation, run by Kyoto University, Graduate School of Medicine at 1 site in Japan.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 3 inclusion and 0 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 22 Sep 2025. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in Japan. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Average time spent creating each document (time frame: On one study day within 2 weeks after enrollment). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT07187050, which also lists 1 linked publication.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 22 Sep 2025.