Assessment of a Novel Algorithm System for Polypharmacy Patient Medication Safety and Consultation

This is an observational, retrospective study performed on deidentified electronic health records (EHRs). The study includes two stages: A pilot study for training and verification of a novel system; and a clinical validation study for evaluating the MDI system performance in comparison to the clinical pharmacist.

The main aim of the study is to determine the noninferiority of a novel system performance in comparison to the clinical pharmacist, for the creation of insights based on the analysis of multidrug patients' EHRs. The quality and accuracy of the analysis and insights is adjudicated by an external judging committee (the gold standard), comprised of two (2) senior academic clinical pharmacists (Pharm.D, Ph.D).

The clinical site's Information systems division scanned for EHRs of multidrug patients according to extraction criteria. The raw data of the patients' records were reviewed by the principal investigator (PI). Medical coding definitions of the extraction criteria were modified according to discrepancies found by the PI. The patients' deidentified EHRs were randomized to a training cohort for the training phase and to the clinical validation cohort.

Stage 1 of the trial was for training the MDI system on real-life patient EHRs. Deidentified EHRs of the training cohort were uploaded to the MDI System platform, which calculates each patient's risk for drug induced problems according to five categories: 1) duplication of therapy, 2) age-related issues, 3) incorrect dose, 4) current side effects and 5) future side effects risks. The MDI System output report is reviewed by the clinical team. The algorithms are modified based on the system-wide conclusions and the risk calculation model.

Stage 2 of the trial was for the clinical validation of the MDI system. The MDI system and clinical pharmacists independently evaluated the clinical cohort EHRs and reported insights on conclusion sheets according to the five categories: 1) duplication of therapy, 2) age-related issues, 3) incorrect dose, 4) current side effects and 5) future side effects risks. An external judging committee comprised of two (2) senior clinical pharmacy doctors, who were blinded to the source of the conclusion sheets, adjudicated the MDI system and clinical pharmacist insights.

Agreement is defined as an identical conclusion between the MDI system/clinical pharmacist's conclusion on any of the categories / problems documented on the patient's conclusion sheet and the judging committee conclusion (the gold standard). The judges were also instructed to add any finding that was missing in their professional opinion from the documented patient conclusion sheets. Agreement of a conclusion with the judging committee's conclusion is marked as true positive. Nonagreement of a conclusion and the judging committee's conclusion is marked as false positive. A conclusion variable found missing by the judging committee is marked as false negative. True negatives are defined as agreement between a conclusion regarding a missing finding in one of the categories.

The sample size for the clinical validation of the MDI System was based on the primary endpoint of non-inferiority of the diagnostic total agreement of the MDI System compared to the diagnostic total agreement of the Clinical Pharmacists for the total of the Combined Categories. The gold standard was the Judging Committee. Assuming that the total agreement is 5% higher for the MDI System, with a non-inferiority margin of 5%, α level of 5%, statistical power of 90%, and an expected standard deviation of 15%, the minimum sample size is about 20 cases. The achieved recruitment level was more than twice as much in the actual clinical trial.

Confidence intervals of 95% (95%CI) were calculated according to the T distribution under the non-inferiority assumption that the total agreement of the MDI algorithm system will not be inferior than the agreement accuracy of the clinical pharmacist. If the 95% CI limit of the percentage differences did not exceed 5% in favor of the clinical pharmacist, the non-inferiority hypothesis was confirmed. The calculation was performed for the primary endpoint (total of the five categories) and the secondary endpoints for each individual category.

The discrete variables assessed in the categories: duplication of therapy, age-related issues and incorrect dose were medications. The discrete variables assessed in the categories: current side effects and future side effects were the individual side effects. Diagnostic accuracy of the MDI System and the Clinical Pharmacists were assessed by the Judging Committee (the "gold standard").

For each subject, total agreement, PPV, NPV, sensitivity and specificity were calculated for all the combined categories together and for each separate category. The diagnostic accuracy of the total subjects is presented as mean, 95%CI of the mean, standard deviation (Std), median, minimum and maximum.

In addition, overall by-line (regardless of subject) validity parameters (overall agreement, PPV, NPV, Sensitivity and Specificity) were analyzed using frequency, percentages and 95% CI.