Clinical DeteriorationReSCUE-MENCT04026555
Realtime Streaming Clinical Use Engine for Medical Escalation
A study testing MEWS++ Monitoring and Predictor Score in people with Clinical Deterioration and Hospital Medicine, run by Icahn School of Medicine at Mount Sinai at 1 site in United States.
Trial summary
The escalation of care for patients in a hospitalized setting between nurse practitioner managed services, teaching services, step-down units, and intensive care units is critical for appropriate care for any patient. Read more about this study
- Who is it for?
- Ages 18 years and older
- Conditions the study is for
- Clinical Deterioration, Hospital Medicine, Monitoring, Physiologic
- What is being tested?
- MEWS++ Monitoring and Predictor Score
- Could I receive a placebo?
- Yes, some participants may receive a placebo
- How many people
- 2,780 participants
- Where
- 1 siteUnited States
- Study dates
- 18 Jun 2019to 19 Mar 2020
- Phase
- Interventional
- Are there results?
- Yes, posted in the registrySummary in the doctors' view
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages 18 years and older
- All sexes
- No healthy volunteers
- Adult, Older adult
Who may be able to join
- All patients age 18 or greater who were admitted to a general care unit selected for each arm.
Who cannot join
- Any admitted patient who has a "Do Not Resuscitate (DNR)" and/or a "Do Not Intubate (DNI)" order in the EHR,
- any patient made "level of care" by RRT as documented in REDCap.
Trial locations and contacts
1 site in 1 country.
About this study
The escalation of care for patients in a hospitalized setting between nurse practitioner managed services, teaching services, step-down units, and intensive care units is critical for appropriate care for any patient. Often such "triggers" for escalation are initiated based on the nursing evaluation of the patient, followed by physician history and physical exam, then augmented based on laboratory values. These "triggers" can enhance the care of patients without increasing the workload of responder teams. One of the goals in hospital medicine is the earlier identification of patients that require an escalation of care. The study team developed a model through a retrospective analysis of the historical data from the Mount Sinai Data Warehouse (MSDW), which can provide machine learning based triggers for escalation of care (Approved by: IRB-18-00581). This model is called "Medical Early Warning Score ++" (MEWS ++). This IRB seeks to prospectively validate the developed model through a pragmatic clinical trial of using these alerts to trigger an evaluation for appropriateness of escalation of care on two general inpatients wards, one medical and one surgical. These alerts will not change the standard of care. They will simply suggest to the care team that the patient should be further evaluated without specifying a subsequent specific course of action. In other words, these alerts in themselves does not designate any change to the care provider's clinical standard of care. The study team estimates that this study would require the evaluation of ~ 18380 bed movements and approximately 30 months to complete, based on the rate of escalation of care and rate of bed movements in the selected units.
Read the detailed description
Objectives:
Mount Sinai Hospital has developed a Rapid Response Team (RRT) system designed to give general floor care providers additional support for patients who may be requiring a higher level of care. This system enables both nurses and physicians to notify the RRT and have a critical care team evaluate the patients. During the period of 03/01/2018 to 09/17/2018, Mount Sinai Hospital floor units on 10W and 10E units made 357 rapid response team (RRT) calls with only 58 leading to an actual increase in the level of care (true positive rate ~ 16%). Similarly, the Electronic Health Record (EHR) generated 839 sepsis Best Practice Alerts (BPAs) yet only five led to escalations in care (true positive rate ~ 0.5%). The results above would imply that over 168 evaluations need to be made to identify a single case where the patient required an escalation in care. The goal of ReSCUE-ME is to evaluate prospective model performance and identify the best spot which the study team can incorporate MEWS++ into RRT and Primary providers workflow. The primary endpoint is rate of escalation of care on 10W and 10E during the study period.
Background:
In a prior study, the group has demonstrated that a machine learning model (MEWS++) significantly outperformed a standard, manually calculated MEWS score on a large retrospective cohort of hospitalized patients. To develop this model, the study team used a data set (Approved by the Program for Protection of Human Subjects Institutional Review Board (IRB) IRB-18-00581) of 96,645 patients with 157,984 hospital encounters and 244,343 bed movements. The study team found that MEWS++ was superior to the standard MEWS model with a sensitivity of 81.6% vs. 44.6%, specificity of 75.5% vs. 64.5%, and area under the receiver operating curve of 0.85 vs. 0.71.
Encouraged by this prior result, the study team is seeking to evaluate the model in a prospective study.
A silent pilot of the ReSCUE-ME alerts has been running on 10E and 10W since Feb 2019. The study team has continuously monitoring the alert performance via a real-time web-based dashboard. The results are summarized below:
- Median # of alerts to primary team, per floor, per day: 8
- Median # of alerts to RRT, per floor, per day: 4
- Sensitivity 0.76, Specificity 0.68, AUC 0.77
- Accuracy 0.69, Precision 0.3, F1 Score 0.43 This performance compares very favorably to the performance seen in the retrospective historical cohort used to develop the MEWS++ model:
- Sensitivity 0.82, Specificity 0.76, AUC 0.85
- Accuracy 0.76, Precision 0.12, F1 Score 0.19"
- Being tested
- MEWS++ Monitoring, Predictor Score
- Keywords
- Medical Early Warning Systems, Patient Monitoring, Electronic Health Record, Big Data, Critical Care, Machine Learning
- Medical subject headings
- Clinical Deterioration
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
MEWS++ MonitoringOther
Patient's electronic medical record data will undergo processing by a machine learning algorithm (MEWS++).
Predictor ScoreOther
A score predicting the likelihood that the patient will experience a deterioration in their clinical condition within six hours will be generated. If the prediction score exceeds a predetermined threshold, an alert will be sent to the provider. The alerting protocol is tiered, with both a low and high threshold. If the score is above the low threshold, nursing will be notified. If the score is above the high threshold, RRT will be notified.
| Study group | Role | What they receive |
|---|---|---|
| MEWS++ Monitoring | Active comparator | This consists of all the patients that will be receiving MEWS++ escalation monitoring and provider alerting. Receives: Other: MEWS++ Monitoring; Other: Predictor Score |
| Standard of Care Monitoring | Placebo comparator | Patients in the control arm will have a score calculated but no alert will be sent. Receives: Other: Predictor Score |
What is being measured
The primary outcome is the main question the study is designed to answer.
Overall Rate of Escalation
Rate of escalation of care from floor to Stepdown, Telemetry, ICU, per 1,000 patient bed days.
Measured: 10 months
Secondary outcomes (7)
Number of Participants Requiring Blood Pressure Support
Number of participants requiring blood pressure support agents such as initiation of vasopressor medication or administration of fluid bolus.
Measured: 10 months
Number of Participants Requiring Respiratory Support
Number of participants requiring respiratory support intervention such as initiation of nasal cannula to high flow or frequency of intubation.
Measured: 10 months
Number of Participants Who Experienced a Cardiac Arrest Episode
The number of patients who had a cardiac arrest.
Measured: 10 months
Mortality Rate
Number of Mortalities - Combined In-hospital and 30-day mortality. Mortalities only counted once. 30-day mortality includes those patients who died in-hospital within 30 days.
Measured: Duration of hospital stay, until discharge, regardless of stay length for patients who died in hospital, or 30 days after admission, starting from date of admission, up to 6 weeks.
Notification Frequency - Number of Alerts Sent Per Day to Providers
The number is calculated as average number of alerts sent per day over all the days in the study period.
Measured: 10 months
Number of Calls
The average number of calls to RRT made per patient, regardless of alert. Not evaluated.
Measured: 10 months
Sensitivity and Specificity of the RRT Alert
The performance of the alert will be evaluated by calculating the sensitivity, specificity, positive predictive value, negative predictive value, precision, recall, and F1-score. This will be done both for the overall escalation rate and if possible for individual escalations (ICU, step-down, telemetry) and death.
Measured: 10 months
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted19 Jul 2019
- Study start18 Jun 2019
- Primary completion19 Mar 2020
- Study completion19 Mar 2020
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Posted resultsParticipant flow, outcome measures and adverse events reported by the sponsor.
- Publications (1)Papers the sponsor has linked to this study.
- Other trials for Clinical DeteriorationStudies of the same condition, open and under way.
- More trials run by Icahn School of Medicine at Mount SinaiThe sponsor's other registered studies.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Prevention
- Allocation
- Non-randomised
- Intervention model
- Parallel groups
- Model description
- For each patient, real-time data from clinical and administrative systems will be used by ReSCUE-ME to produce a MEWS++ score predicting the likelihood that the patient will require escalation of care within the next 6 hours. Upon the patient being admitted to the unit, the patient will be evaluated based on any update in the EHR. If the prediction score exceeds a "high" threshold, the RRT team will be notified directly. If the score is between a "low" threshold and the high threshold , the nursing team will be notified and increased nursing monitoring will be initiated. If the patient has met criteria for increased nursing monitoring, a refractory 8-hour refractory window will be applied during which no nursing alerts will be sent. However if the score exceeds the high threshold, the RRT team will be notified. Throughout the trial, the performance of the alerts will be monitored via web-based dashboards. If the performance is poor, the "high" and "low" thresholds will be adjusted.
- Masking
- None (open label)
- Enrolment
- 2,780 participants (actual)
Sponsor and investigators
- Lead sponsor
- Icahn School of Medicine at Mount Sinai
- Sponsor type
- University, hospital or other
- Responsible party
- Principal investigator, Matthew Levin, Icahn School of Medicine at Mount Sinai
- Investigators
- Matthew A Levin, MD, Study director, Icahn School of Medicine at Mount Sinai
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT04026555
- Sponsor's protocol ID
- GCO 19-0729
- Acronym
- ReSCUE-ME
- Status last verified
- Jan 2025
- First submitted
- 16 Jul 2019
- Last updated
- 14 Jan 2025
- Results posted
- Yes, on ClinicalTrials.gov
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- Undecided: Deidentified raw data may be shared with other researchers upon submission of a written request along with a plan for how the data are intended to be used.
Publications
Levin MA, Kia A, Timsina P, Cheng FY, Nguyen KA, Kohli-Seth R, Lin HM, Ouyang Y, Freeman R, Reich DL. Real-Time Machine Learning Alerts to Prevent Escalation of Care: A Nonrandomized Clustered Pragmatic Clinical Trial. Crit Care Med. 2024 Jul 1;52(7):1007-1020. doi: 10.1097/CCM.0000000000006243. Epub 2024 Feb 21.
PubMed 38380992
Study at a glance
From the registry record, last updated by the sponsor on 14 Jan 2025. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Realtime Streaming Clinical Use Engine for Medical Escalation
- ClinicalTrials.gov ID
- NCT04026555
- Acronym / sponsor ID
- ReSCUE-ME / GCO 19-0729
- Keywords
- Medical Early Warning Systems, Patient Monitoring, Electronic Health Record, Big Data, Critical Care, Machine Learning
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 2,780 participants (actual)
- Lead sponsor
- Icahn School of Medicine at Mount SinaiUniversity, hospital or other
- Study start
- 18 Jun 2019 (actual)
- Primary completion
- 19 Mar 2020 (actual)
- Study completion
- 19 Mar 2020 (actual)
- First posted
- 19 Jul 2019 (actual)
- Results
- Posted 14 Jan 2025
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
The escalation of care for patients in a hospitalized setting between nurse practitioner managed services, teaching services, step-down units, and intensive care units is critical for appropriate care for any patient. Often such "triggers" for escalation are initiated based on the nursing evaluation of the patient, followed by physician history and physical exam, then augmented based on laboratory values. These "triggers" can enhance the care of patients without increasing the workload of responder teams. One of the goals in hospital medicine is the earlier identification of patients that require an escalation of care. The study team developed a model through a retrospective analysis of the historical data from the Mount Sinai Data Warehouse (MSDW), which can provide machine learning based triggers for escalation of care (Approved by: IRB-18-00581). This model is called "Medical Early Warning Score ++" (MEWS ++). This IRB seeks to prospectively validate the developed model through a pragmatic clinical trial of using these alerts to trigger an evaluation for appropriateness of escalation of care on two general inpatients wards, one medical and one surgical. These alerts will not change the standard of care. They will simply suggest to the care team that the patient should be further evaluated without specifying a subsequent specific course of action. In other words, these alerts in themselves does not designate any change to the care provider's clinical standard of care. The study team estimates that this study would require the evaluation of ~ 18380 bed movements and approximately 30 months to complete, based on the rate of escalation of care and rate of bed movements in the selected units.
Detailed description
Objectives:
Mount Sinai Hospital has developed a Rapid Response Team (RRT) system designed to give general floor care providers additional support for patients who may be requiring a higher level of care. This system enables both nurses and physicians to notify the RRT and have a critical care team evaluate the patients. During the period of 03/01/2018 to 09/17/2018, Mount Sinai Hospital floor units on 10W and 10E units made 357 rapid response team (RRT) calls with only 58 leading to an actual increase in the level of care (true positive rate ~ 16%). Similarly, the Electronic Health Record (EHR) generated 839 sepsis Best Practice Alerts (BPAs) yet only five led to escalations in care (true positive rate ~ 0.5%). The results above would imply that over 168 evaluations need to be made to identify a single case where the patient required an escalation in care. The goal of ReSCUE-ME is to evaluate prospective model performance and identify the best spot which the study team can incorporate MEWS++ into RRT and Primary providers workflow. The primary endpoint is rate of escalation of care on 10W and 10E during the study period.
Background:
In a prior study, the group has demonstrated that a machine learning model (MEWS++) significantly outperformed a standard, manually calculated MEWS score on a large retrospective cohort of hospitalized patients. To develop this model, the study team used a data set (Approved by the Program for Protection of Human Subjects Institutional Review Board (IRB) IRB-18-00581) of 96,645 patients with 157,984 hospital encounters and 244,343 bed movements. The study team found that MEWS++ was superior to the standard MEWS model with a sensitivity of 81.6% vs. 44.6%, specificity of 75.5% vs. 64.5%, and area under the receiver operating curve of 0.85 vs. 0.71.
Encouraged by this prior result, the study team is seeking to evaluate the model in a prospective study.
A silent pilot of the ReSCUE-ME alerts has been running on 10E and 10W since Feb 2019. The study team has continuously monitoring the alert performance via a real-time web-based dashboard. The results are summarized below:
- Median # of alerts to primary team, per floor, per day: 8
- Median # of alerts to RRT, per floor, per day: 4
- Sensitivity 0.76, Specificity 0.68, AUC 0.77
- Accuracy 0.69, Precision 0.3, F1 Score 0.43 This performance compares very favorably to the performance seen in the retrospective historical cohort used to develop the MEWS++ model:
- Sensitivity 0.82, Specificity 0.76, AUC 0.85
- Accuracy 0.76, Precision 0.12, F1 Score 0.19"
Study design
- Primary purpose
- Prevention
- Allocation
- Non-randomised
- Intervention model
- Parallel groups
- Masking
- None (open label)
- Number of arms
- 2
- Model description
- For each patient, real-time data from clinical and administrative systems will be used by ReSCUE-ME to produce a MEWS++ score predicting the likelihood that the patient will require escalation of care within the next 6 hours. Upon the patient being admitted to the unit, the patient will be evaluated based on any update in the EHR. If the prediction score exceeds a "high" threshold, the RRT team will be notified directly. If the score is between a "low" threshold and the high threshold , the nursing team will be notified and increased nursing monitoring will be initiated. If the patient has met criteria for increased nursing monitoring, a refractory 8-hour refractory window will be applied during which no nursing alerts will be sent. However if the score exceeds the high threshold, the RRT team will be notified. Throughout the trial, the performance of the alerts will be monitored via web-based dashboards. If the performance is poor, the "high" and "low" thresholds will be adjusted.
| Arm | Type | Receives |
|---|---|---|
| MEWS++ Monitoring | Active comparator | Other: MEWS++ Monitoring, Other: Predictor ScoreThis consists of all the patients that will be receiving MEWS++ escalation monitoring and provider alerting. |
| Standard of Care Monitoring | Placebo comparator | Other: Predictor ScorePatients in the control arm will have a score calculated but no alert will be sent. |
Who may qualify
- Ages
- 18 years and older
- Sex
- All
- Healthy volunteers
- Not accepted
- Age groups
- Adult, Older adult
Key inclusion criteria (1 listed)
- All patients age 18 or greater who were admitted to a general care unit selected for each arm.
Key exclusion criteria (2 listed)
- Any admitted patient who has a "Do Not Resuscitate (DNR)" and/or a "Do Not Intubate (DNI)" order in the EHR,
- any patient made "level of care" by RRT as documented in REDCap.
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| MEWS++ MonitoringPatient's electronic medical record data will undergo processing by a machine learning algorithm (MEWS++). | Other | MEWS++ Monitoring |
| Predictor ScoreA score predicting the likelihood that the patient will experience a deterioration in their clinical condition within six hours will be generated. If the prediction score exceeds a predetermined threshold, an alert will be sent to the provider. The alerting protocol is tiered, with both a low and high threshold. If the score is above the low threshold, nursing will be notified. If the score is above the high threshold, RRT will be notified. | Other | MEWS++ Monitoring, Standard of Care Monitoring |
Primary outcome
Overall Rate of Escalation
Time frame: 10 months
Rate of escalation of care from floor to Stepdown, Telemetry, ICU, per 1,000 patient bed days.
7 secondary outcomes
Number of Participants Requiring Blood Pressure Support
Time frame: 10 months
Number of Participants Requiring Respiratory Support
Time frame: 10 months
Number of Participants Who Experienced a Cardiac Arrest Episode
Time frame: 10 months
Mortality Rate
Time frame: Duration of hospital stay, until discharge, regardless of stay length for patients who died in hospital, or 30 days after admission, starting from date of admission, up to 6 weeks.
Notification Frequency - Number of Alerts Sent Per Day to Providers
Time frame: 10 months
Number of Calls
Time frame: 10 months
Sensitivity and Specificity of the RRT Alert
Time frame: 10 months
Results and safety
Loading the posted results from ClinicalTrials.gov…
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 14 Jan 2025.
- Posted resultsParticipant flow, baseline, all outcome measures and adverse events.
- Derived from this study: PubMed 38380992Levin MA, Kia A, Timsina P, Cheng FY, Nguyen KA, Kohli-Seth R, Lin HM, Ouyang Y, Freeman R, Reich DL. Real-Time Machine Learning Alerts to Prevent Escalation of Care: A Nonrandomized Clustered Pragmatic Clinical Trial. Crit Care Med. 2024 Jul 1;52(7):1007-1020. doi: 10.1097/CCM.0000000000006243. Epub 2024 Feb 21.
- Published articleLink provided by the sponsor.
- Individual participant dataSharing planned: Undecided. Deidentified raw data may be shared with other researchers upon submission of a written request along with a plan for how the data are intended to be used.
Related trials for Clinical Deterioration
Recruiting
The Orientation Intervention to Rapid Response Teams (ORIENT-RRT) Study: A Quasi-Experimental Study
Deterioration, Clinical
Not yet recruiting
MAGICS - Multi-Agent Generative AI in Clinical Simulation for Clinical Deterioration Training
Clinical Reasoning in Nursing Education, Virtual Simulation
RecruitingObservational study
The ReTAVI Prospective Observational Registry
Structural Valve Deterioration, Structural Valve Degeneration
RecruitingObservational study
Repeated Examinations for Typing Pulmonary Embolism
Pulmonary Embolism (Diagnosis), Pulmonary Embolism Acute
Not yet recruitingObservational study
Development and Internal Validation of a Multimodal Prediction Score for Early Hemodynamic Deterioration in Adult Polytrauma Patients
Multiple Trauma, Polytrauma
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
Matthew A Levin, MD study director
Icahn School of Medicine at Mount Sinai
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing MEWS++ Monitoring and Predictor Score in people with Clinical Deterioration and Hospital Medicine, run by Icahn School of Medicine at Mount Sinai at 1 site in United States.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 1 inclusion and 2 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 14 Jan 2025. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in United States. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. Some participants in this study may receive a placebo. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
Yes. Results are posted in the registry (first posted 14 Jan 2025). A summary is in the For doctors view, and the full results are on ClinicalTrials.gov.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Overall Rate of Escalation (time frame: 10 months). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT04026555, which also lists 1 linked publication.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 14 Jan 2025.