Comparison of Two Artificial Intelligence Scribe Products on Pediatric Subspecialty Provider Wellness and Experience, Patient Satisfaction, and Efficiency.

Background: While electronic health record (EHR) systems have contributed to advances in patient safety and quality of care, they have also been associated with a significant increase in documentation burden, contributing to burnout among clinicians. This is particularly true for physicians with insufficient time for documentation. In some cases, it has resulted in a reduction in appointment slots to allow for additional documentation time, which in turn decreases patient access to care and physician productivity.

Artificial intelligence (AI) scribes use visits recorded with verbal patient/parental consent and leverage generative AI to create note sections in near-real time that the provider can use and edit as they see fit. In addition, it allows providers to continue to use their documentation templates while adding the generative AI to "smart sections" within their note. This approach has the potential to substantially reduce documentation burden while maintaining documentation preferences of many providers

. This rapid, randomized quality improvement trial aims to assess whether the implementation of generative AI software for documentation can enhance physician documentation efficiency and reduce burnout. It also aims to determine which of two vendors is most effective overall and cost-effective for the health system.

Objectives Quality Improvement Global Aim: To increase provider documentation efficiency and reduce provider burnout related to documentation burden.

Children's Operational Goal: Determine if the cost of ambient AI scribe products (DAX Copilot and Abridge) is justified by reduction in proxies for physician burnout and/or could be offset by seeing more patients in the same time period to improve revenue and patient access.

Goals of the Proposed Work:

• Determine the influence of ambient AI scribe on proxies for physician burnout including pajama time, time in notes, and subjective measures of EHR efficiency.
• Once providers understand the impact of ambient AI scribe on their personal workflow, quantify their willingness to see more patients to offset the costs of the software if needed to subsidize the costs of the technology.
• Determine which AI scribe vendor has the most impact on measures above.

Expected Next Steps: If ambient AI scribe users have significant and substantial improvement in pajama time, time in notes, and subjective measures of EHR efficiency, then the organization will likely aim to expand the subscription for this or related vendor software and implement more broadly with Children's Physician Group providers. However, if no significant differences are observed, we will reconsider the use of this software and assess alternative approaches to address documentation-related challenges.

Methods:

We will assess changes to proxies for provider documentation efficiency and burnout through a difference-in-differences design, as well as directly compare the efficacy of two different ambient AI scribe products (DAX Copilot and Abridge).

Project Participants

The project will recruit 105 providers on a voluntary basis with the following inclusion criteria/considerations to use ambient AI scribe software:

1. Outpatient practices at Children's in a specialty supported by integrated ambient AI scribe software
2. Includes Urgent Care and Emergency Department
3. Excludes providers working in hospital-based clinics due to lack of availability of workflow integration into the HER
4. Does not currently utilize an in-person human scribe
5. High clinical workload during pilot period
6. Agrees to use the Children's EHR mobile application (Haiku) on their personal device.
7. Agrees to offer use of the ambient AI scribe software for at least 75% of patient visit encounters for the duration of the project period

Twenty providers will be randomized to DAX Copilot (total of 40 including those from a prior phase 1 pilot) and 50 providers to Abridge software in an equal distribution of specialties, with the remaining 35 randomized to a control group (no AI scribe software; continue usual documentation practices). After 1-2 months, 40 of the providers in each intervention cohort will be asked to switch to using the other product for another 1-2 months and provide qualitative comparative feedback on the two products.

Outcomes

The primary outcomes to be obtained through Epic's Signal product will be:

1. "Pajama Time", defined as the average number of minutes per scheduled day spent in charting activities outside 7 AM to 5:30 PM on weekdays, time outside scheduled hours on weekends, and time on unscheduled holidays. This metric is associated with the exhaustion subscale of the Maslach Burnout Inventory.
2. "Time in Notes per Appointment" in minutes Additional outcome metrics will include:
3. Progress Note Length (characters)
4. Note Contribution (written by provider vs. others)
5. Time to Appointment Closure
6. Proportion of notes completed using ambient AI scribe software
7. Average patient volume per week
8. Patient satisfaction
9. Pre- and post-project user responses on a modified KLAS EHR Efficiency and Satisfaction survey or Stanford Professional Fulfillment Index survey, both of which are validated benchmarking tools.
10. Subjective feedback, including comparison between DAX Copilot and Abridge if applicable

Data Collection

Data will be collected from Epic© Signal and through surveys. The data will include:

• Demographic information of project participants
• Data related to documentation efficiency as outlined in the quality metrics above

All patients seen at Children's are asked to complete a patient satisfaction survey as part of usual processes; this survey is sent asynchronously via an email that is typically sent after the medical visit has occurred. Due to the asynchronous nature of this survey, patient response rate is typically quite low.

To ensure an adequate number of patient responses for statistical analysis, we will administer the same patient satisfaction survey synchronously (at the end of the clinic visit before the patient departs) for 5 patients pre-intervention and 5 patients post-intervention for both intervention and control. The only change from usual operational practice would be the timing of survey administration.

Statistical Analysis

The primary analysis will be a difference-in-differences analysis for each outcome. For example, the difference between the provider's average pajama time before and after the intervention period will be calculated for all participants. The investigators will then determine how this average differs in the AI group and in the control group to assess the difference-in-differences.

Additional analyses will include adjusted or stratified difference-in-differences analyses based on provider characteristics listed above. The investigators will also calculate descriptive statistics to compare the outcomes and covariates between the two groups. Depending on the nature of the data, the investigators may use run charts, t-tests, ANOVA, or other appropriate statistical methods to assess the impact of generative AI documentation software.