Comparing Clinical Decision-making of AI Technology to a Multi-professional Care Team in ECBT for Depression

Participants (n = 186: n = 31 per e-CBT group * 2 arms) will be recruited at Queen's University from outpatient psychiatry clinics at both Kingston Health Sciences Centre sites (Hotel Dieu Hospital and Kingston General Hospital), as well as Providence Care Hospital in Kingston, Ontario. Additionally, self-referrals and referrals from family doctors, physicians, and clinicians across Ontario will be accepted. After obtaining informed consent from the participant, the participant will be evaluated using the Mini International Psychiatric Assessment (MINI) through a secure video appointment to confirm a diagnosis of Major Depressive Disorder using the DSM-5 by a trained professional on the research team.

All eligible participants will be randomized to receive a treatment plan based on the decision of either the healthcare team (Arm 1) or the Triage Module using an AI algorithm (Arm 2). Participants will be randomly allocated to one of the two arms of the study by a research assistant on the team who will also balance the group based on demographic variables (i.e., sex, gender, age, and income). Participants and therapists in the study will be blinded to which treatment arm the participant belongs to. By the nature of this study, participants and therapists will not be blinded to which treatment intensity the participant will receive since it will be evident whether the participant is receiving a phone/video call in addition to usual e-CBT care or pharmacotherapy. Each participant will be provided with an effective form of treatment (i.e., e-CBT) regardless of which group they will be allocated to. Participants will be informed that there is no incentive for joining the program and that joining or withdrawing at any point will not affect them negatively. It will also be explained to the participants that the program is not a crisis resource and that they will not always have access to their therapists. In the case of an emergency, participants will be directed to proper resources, and this event will be reported to the study's lead psychiatrist (principal investigator). All data will be anonymized and will be analyzed by members of the research team who are not directly involved in the patient's care.

Treatment Arm 1: Healthcare Team Allocation

Allocation of treatment intensity by the multi-professional healthcare team will be based on the following criteria:

1. The severity of MDD symptoms (using DSM-5 criteria).
2. Mental health factors (prior treatments and responses, current and past psychotic/manic episodes, current and past suicidal/homicidal ideation/attempts, family mental health history, past psychiatric history, and hospital admissions).
3. Medical factors (current medical conditions and medications, personal and family medical history).
4. Social factors (support system and living situation, and occupational, social, and personal functional impairment).

Additionally, to assess the severity of MDD symptoms and the functional impairments, participants will complete the PHQ-9 and Sheehan Disability Scale (SDS) before the assessment appointment. The assessment appointment will be conducted by the trained research assistant on the multidisciplinary team who will relay the information to the rest of the team later to deliberate on treatment intensity allocation. All assessments will occur virtually through phone and video calls. Together, the healthcare team will decide whether the participant should be assigned to the e-CBT-only treatment, e-CBT treatment with weekly phone/video calls, or e-CBT treatment with pharmacotherapy. This process mimics the current triage process in clinical settings. To track cost-effectiveness, the trained research assistant will track the total duration of the individual assessment and team deliberation meetings for analysis of the total time commitment per patient.

Treatment Arm 2: AI Algorithm Allocation Allocation of treatment intensity by the proposed AI algorithm will be based on the machine learning and natural language processing (NLP) of textual data provided by participants and their PHQ-9 score collected through a pre-treatment screening module called the Triage Module. This module, developed by the research team, (1) provides psychoeducation on the effects of psychotherapy, (2) collects PHQ-9 scores, and (3) asks participants six open-ended questions regarding their mental health history, their experiences with mental health disorders, and what mental health difficulties they are currently facing. Based on the participant's answers to the open-ended questions, a variable called "Symptomatic Score" will be calculated using the NLP algorithm. If the PHQ-9 score < 19 and the Symptomatic Score > 0.75, the participant will be assigned to the e-CBT-only treatment group. However, if either the PHQ-9 score is > 19 or the Symptomatic Score is < 0.75, the participants will be assigned to the e-CBT treatment with weekly phone/video calls. If both scenarios occur and the PHQ-9 > 19 and Symptomatic Score < 0.75, then the participant will be assigned to the e-CBT treatment with pharmacotherapy.

To gather the relevant data (i.e., participant compliance and change in depression severity, as evaluated by the PHQ-9), the triage module was designed. As previously explained, NLP of the participants' written accounts of their challenges with depression in the Triage Module will be used to calculate a Symptomatic Score. To verify the AI's treatment allocation logic, the completion rate and the change in PHQ-9 scores were assessed in a sample of participants (n = 190) who were previously enrolled in e-CBT-only treatment. The decision-making algorithm determined that the e-CBT-only program was suitable for 62 out of the 190 participants (33%). Within these 62 participants, 54% had completed the e-CBT-only program in its entirety and only 20% had a final PHQ-9 score > 14. Furthermore, the algorithm indicated that e-CBT with telephone calls would be suitable for 100 out of the 190 participants (53%). Of the 100 participants, 41% completed the whole round of e-CBT-only therapy and 31% had a final PHQ-9 score > 14. Lastly, the algorithm indicated that e-CBT with video call was appropriate for 28 out of 190 participants (14%). Of these 28 participants, 35% completed the whole round of e-CBT-only therapy and 40% had a final PHQ-9 score > 14. The logic of the AI's decision is therefore justified as those participants allocated to the e-CBT-only group had the highest percentage of completion and lowest percentage of final PHQ-9 scores > 14 when completing e-CBT-only. Therefore, these individuals require minimal therapist intensity, and e-CBT-only is sufficient. Conversely, participants allocated to the e-CBT with video call had the lowest completion rates and highest rates of final PHQ-9 scores > 14 when enrolled in e-CBT-only. These findings justify the AI's logic that greater therapist interaction is required. It is also important to note that demographic factors like age (below or above 40 years), sex (male or female) and income (less or more than $50K) did not have any significant effects on the number of sessions completed by participants (p = 0.92, 0.18 \& 0.9 for age, sex, and income respectively). The demographic factors did not affect the change in PHQ-9 score (i.e., the difference between the beginning and end of treatment scores) either (p = 0.2, 0.46 \& 0.39 for age, sex, and income respectively).

e-CBT Program The e-CBT sessions used in this study include content based on cognitive restructuring and behavioural activation techniques. The purpose of the sessions is to help participants become aware of inaccurate or negative thinking patterns so that they can view challenging situations more clearly and respond to them effectively. The sessions prompt participants to understand their situation/environment and the resulting thoughts, behaviours, physical reactions, and feelings. The goal of this program is to help change participants' negative and/or ineffective thoughts to more effective ways of thinking. As expressed in CBT, changing thoughts can subsequently affect feelings, behaviours, and physical reactions to stressful situations.

Therapists Each participant will be assigned a care provider that will provide feedback for their weekly sessions before the start of their next session. The assigned care provider will be independent of the multi-professional healthcare team that conducted the intake assessment. All care providers are trained in psychotherapy and have experience delivering electronic psychotherapy. They will be informed of the aim and the content of each therapeutic session. They will also continue receiving specialized training through webinars, workshops and exercises with feedback provided by the lead psychiatrist on the research team, a trained and licensed psychotherapist. All care providers will be supervised by a trained psychotherapist and the lead psychiatrist, and all feedback will be reviewed before submission to the participants.

e-CBT Weekly Feedback Weekly homework is reviewed by the independent care provider assigned to the participant, who will provide text-based personalized feedback on OPTT before the next weekly session. Additionally, the participants and care providers can communicate asynchronously on OPTT to relay any questions or concerns. The care providers will be provided with sample feedback templates and scripts for the telephone and video call sessions. Templates and scripts will be adapted from previous studies conducted by the research team. Feedback templates and scripts will vary between sessions, and care providers will personalize them for each patient. The feedback templates follow a generic structure starting with, acknowledging the participant's time and effort since the last session, summarising the CBT concepts taught in the previous session, reviewing the event they explained in their homework, validating the participant's experience(s), and encouraging the participant to keep up with the sessions. The feedback is written in a letter format to increase personalization and build rapport with the participants.