Evaluating a Virtual Mentoring Program Plus Mental Health Videos for Teens

Pilot Study (1 year). Optimize Study Procedures and Participant Retention (6 months). In an initial pilot study with N=75 youth, the researchers will examine the adapted Appa Health and research study content, offering data for optimization and retention incentive decisions prior to larger scale RCT through the following research questions (RQs): RQ 1.1: Are research and incentive protocols sufficient for recruiting and retaining adolescents in the study, and how can they be optimized? RQ 1.2. Do Appa's modified engagement strategies developed in preliminary studies increase use of CBT video content? This pilot study will enable the research team to identify and address any problems in recruitment, study procedures, and retention prior to the effectiveness trial.

Setting. The study will be carried out within the Appa Health remote platform, which comprises over 20 trained youth mentors serving, on average, 400 youth every 3 months. Clients come primarily from school-based partnerships. Appa has several existing school partners who refer students to Appa either through a paid partnership from the school or through access to parents of students.

Recruitment and Randomization. Participants will be recruited from the Appa Health client population using email outreach. As described in the Commercialization Plan, the Appa client base from individual paying clients, school systems, and B2B customers is expected to be sufficient to meet this goal within the study timeframe. Screening of youth using the PHQ-8 and GAD is standard protocol for Appa; those who score above 10 on either instrument will be invited to participate in the study. Due to the relatively small sample size of the pilot trial, the research team will use stratified random sampling to ensure approximate balance to the three study conditions on baseline covariates (gender, age, baseline severity on GAD \& PHQ-8, race/ethnicity, and LGBTQ status), with stratification blocks created via nearest neighbor matching on all variables.

Interventions. Youth will be randomized to three conditions, which in the larger RCT will enable analyses of the main effect of the traditional Appa package as compared to a waiting list control group, as well as exploring the impact of a light touch version of Appa for commercial purposes. This will also permit examination of the impact of the two components on specific psychosocial mechanisms, as appropriate in an experimental therapeutics paradigm.

"Appa Lite": Unguided EBP videos and materials. Participants assigned to the Appa Lite condition will have unlimited access to the Appa website, including short form CBT-based videos and materials such as an emotions wheel pdf and thoughts, feelings, and behavior worksheets.

"Appa Complete": Weekly mentorship meetings \& guided CBT videos. Participants assigned to Appa Complete will have unlimited access to the Appa website and materials described in Appa Lite, along with weekly mentorship meetings with a trained and supervised near-peer mentor with whom they have been matched.

"Waiting List Control": Participants assigned to the waitlist control will receive a small pdf packet of psychoeducational materials and will be permitted to use Appa at the end of the 12 weeks of data collection. Families assigned to this condition will not pay for Appa's services while on the waitlist.

Data Collection. The REDCap electronic data collection application will be used to collect data from mentors, supervisors, and caregivers at enrollment (baseline) and at midpoint (6 weeks) and end (12 weeks).REDCap will also be used to collect survey data from adolescents at enrollment (baseline) and at midpoint (6 weeks) and end (12 weeks). Adolescent data collection will include quantitative and qualitative surveys and assessments focused on therapeutic interactions (mentorship fidelity, frequency of video watching and mentor interactions, supportive accountability), mechanisms of change (e.g., use of CBT, confidence in relationships, reduced isolation), clinical outcomes (depression, anxiety, top problems, quality of life/flourishing, self-esteem), possible moderators (demographics, alliance, mentor-adolescent match), and study-related measures (research burden, open-ended comments). To control for compensatory mental health service use by participants in any condition, the prohibition of which in the study would be unethical, the research team will collect data on non-Appa services received elsewhere, including mentorship. Adolescents will be incentivized with $25 per survey timepoint and non-monetary, low burden incentives such as humorous messaging and GIFs. Mentor data collection will focus on mentor fidelity to the supportive accountability model (coded audio recordings of mentor sessions and text communication), moderators (surveys of alliance, mentor-client match), and time diaries for the cost analysis. Mentor supervisors will also provide time diary data. Caregivers will provide data on caregiver strain and the financial burden of adolescent care due to mental health, and caregivers will be incentivized with $5 per administration. Fidelity to mentorship will be measured by coding three randomly selected audio recordings of mentorship sessions and 10 randomly selected text exchanges for each client. Cost assessments. Activity-based costing will estimate the incremental cost of Appa Complete and Appa Lite. The research team will estimate the direct cost of video production and mentoring including web hosting, payments to video creators, mentor salary, training, and supervision. Data collection using time diaries will occur via online surveys of mentors and supervisors. The research team will identify activities related to mentoring and associated labor and non-labor inputs such as time, supplies, travel, overhead, and costs associated with meetings. Activities will be categorized to facilitate understanding of cost allocation. The price of goods and services will be available via budget and expense reports.

Effectiveness Trial: Study Appa Costs and Effectiveness (2.5 years). A large randomized controlled trial with N=400 youth will replicate the pilot study, excluding measures collected on research burden and incentive satisfaction. If the research team does not make substantive changes to the product after the initial pilot trial, the main effectiveness trial analysis will include all 75 pilot participants (total N=up to 475 youth). The three-arm design will enable Appa to determine the cost and effectiveness of each Appa component and enable evidence-based decision making on the commercial viability of scaling these components. Appa will learn the degree to which users engage in components, the cost of those components, and the impact of those components on important mechanisms and outcomes. The research team will test the following hypotheses: H1a-j: At 12 week follow-up, teens in the Appa Complete condition will have significantly greater improvement than both other conditions on primary outcomes of a) youth top problem severity, b) use of CBT skills, c) self-efficacy for CBT; as well as secondary outcomes of youth d) depression, e) anxiety, f) quality of life, g) self-esteem, h) existential isolation, and caregiver i) strain, and h) financial burden. H2a-h: Clients in the Appa Lite condition will have superior outcomes than the control group on a) use of CBT skills, b) depression, c) anxiety, d) top problem severity, e) quality of life, f) self-esteem, g) caregiver strain and h) caregiver financial burden; H3a-d: Near peer mentoring will increase engagement in digital tools and mechanisms such that those who receive mentoring (Appa Complete), when compared to Appa Lite, will a) watch more videos, b) have greater efficacy for CBT skills, c) report greater use of CBT skills, and d) report greater meaning and purpose. Exploratory research questions: ERQ1: Do hypothesized mechanisms (meaning and purpose, identification with mentor, self-efficacy for using CBT, fidelity to practice and supportive accountability, and use of CBT skills) mediate the impact of Appa on depression, anxiety, and top problem severity? ERQ2: Is the impact of Appa on top problem severity, depression, and anxiety moderated by client demographics, mentor alliance, frequency of meetings, degree of demographic match between client and mentor, or dosage of video use? The research team is particularly interested in whether there are differential outcomes by race and social identity.

Cost-effectiveness. H4: Appa Complete will be more expensive but more cost-effective than Appa Lite.

Participants. Inclusion and exclusion criteria will replicate the pilot study.

Recruitment. Recruitment methods will replicate the pilot study. The research team will recruit approximately 400 adolescents, roughly 133 in each condition. The research team assumes 20% attrition for a final complete sample of 320; however, these analytic methods permit inclusion of all available data including dropouts. If the research team does not make substantive changes to the product after the initial pilot trial, the main effectiveness trial analysis will include all 75 pilot participants (total N=up to 475 youth).

Randomization. Random assignment to condition will occur using urn randomization at the client level to ensure roughly equal numbers in each condition based on key hypothesized moderators including gender, age, baseline severity on GAD \& PHQ-8, race/ethnicity, and LGBTQ status. Urn randomization is an adaptive biased-coin method that ensures balance on multiple covariates in rolling-enrollment designs but approaches complete randomization in larger sample sizes.

Data collection. Data collection will replicate the pilot study.