An Evaluation of the Yes You Can... Make Smart Choices Curriculum

Introduction

The New Jersey Physicians Advisory Group (NJPAG) previously created the Yes You Can…Make Smart Choices! (YYC…MSC!) program with the goal of preventing teen pregnancy and enhancing character development in youth. YYC…MSC! is a fully developed intervention that NJPAG has experience implementing that is grounded in a theoretical framework with promising evidence of effectiveness. However, this intervention has not yet undergone a rigorous evaluation with a control or comparison group. This evaluation aims to conduct a randomized control trial (RCT) to determine the program's effectiveness. This evaluation will focus on the entire PREIS planned intervention, YYC…MSC!. The intervention will be implemented in 12 Newark Public Schools in New Jersey. Findings from this study will expand the evidence base on adolescent pregnancy prevention through this innovative program. After the study, findings will be disseminated to encourage program interest, support, and adoption in real-world service systems or communities, including schools.

Intervention Activities

The YYC…MSC! is an 11-lesson curriculum that will be delivered to 9th graders in Newark, New Jersey public schools during 40-minute health classes. The curriculum will be delivered in-person over approximately 2 weeks by NJPAG educators. The goal of the program is to prevent teen pregnancy and enhance character development in youth. The PREIS funding is supporting the implementation and evaluation of the program. Each time an NJPAG educator is in the classroom, they will deliver a single lesson of the curriculum. All NJPAG educators have received internal training for the program; their training includes: (1) studying the curriculum and completing activity/learning pages, (2) practicing teaching to other educator in a mock classroom setting, followed by feedback from the team, (3) observing a seasoned educator teaching the curriculum in its entirety, (4) co-teach the curriculum with a seasoned educator, and (5) teaching on their own, with observation. YYC…MSC! is delivered like a traditional classroom curriculum, including lectures and activities, meaning all students in a given classroom will receive the lectures concurrently. The intervention addresses the following adult preparation subjects: parent-child communication, healthy relationships, healthy life skills, adolescent development, and financial literacy.

Since its inception in 2004, NJPAG has been a non-profit in New Jersey with the goal of addressing the societal problems of teen pregnancy, teen STDs/STIs, and single-parent teen families. To meet this goal, NJPAG has created various curricula to educate youth in New Jersey and beyond, including the YYC…MSC! curriculum for this intervention. NJPAG has a positive, existing relationship with the Newark Public Schools (NPS) Board of Education. As a result of this relationship, NJPAG has the approval to provide the intervention as part of all NPS's 9th-grade health classes. Schools with potentially high rates of language barriers were eliminated because the program is only taught in English. After eliminating schools with potentially high rates of language barriers, 13 NPS schools were selected to participate in the evaluation, five control schools, and five treatment schools.

Internally, NJPAG has a Board of Directors, Executive Director, Program Director, and educators. After receiving training, NJPAG educators will go into Newark public high schools to build rapport with the school's teachers, staff, and students and explain the intervention and related processes. Once rapport is built, NJPAG will go to classes and disseminate the written notice form for youth to bring home to parents to inform parents of the program their child/student will receive and give the parents the option to opt their child/student out of the evaluation. Two weeks after the written notice is disseminated, an Educator will go back to the classroom and collect any forms returned by parents who opted their child out of the evaluation. Upon receiving opt-outs, an educator will facilitate the pre-survey. The following school day after the pre-survey, the educator will go back to the classroom to begin program delivery, starting with lesson 1 and subsequently teaching a lesson every time they go to the classroom. Each lesson addresses a different topic related to pregnancy prevention and has lesson goals.

We expect all lessons to work together to achieve the program goal. For intermediate outcomes, we expect that treatment school youth, e.g., those that receive the intervention, will report an increase in pregnancy prevention knowledge, skills, and confidence and a decrease in activity that could lead to unwanted pregnancy. We believe this intervention will impact youth who received the program long-term by improving life skills, such as emotional coping and sexual, cognitive, and social health, decreasing incidences of unwanted pregnancy, decreasing incidences of STIs, and enhancing character development.

Target Population

NJPAG has been serving New Jersey public schools for almost two decades. Recently, NJPAG was approved to provide sex education programming to 9th-grade health class students in NPS by the Newark Board of Education.

The target population is youth in 9th grade in urban school districts in the Northeast region without any severe cognitive, mental, or behavioral impairments.

The total population of youth who would receive the intervention is approximately 2,400 students. The average expected age of participants is 14. Consistent with the NPS district student population race/ethnicity breakdown, the expected race/ethnicity breakdown of participants is as follows: Black (38.8%), Hispanic (51.0%), White (8.4%), Asian (< 1%), Native Hawaiian/ Pacific Islander (< 1%), American Indian (< 1%), Unspecified (< 1%).

Research Questions

The intervention being tested is the YYC…MSC! program and the target population are youth in 9th grade in urban school districts in the Northeast region without any severe cognitive, mental, or behavioral impairments. The counterfactual condition is business as usual for control schools. The intervention occurs over 11 consecutive school days in health classes.

The primary research question is, "What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of engaging in sexual intercourse 12-months after the pre-survey compared to those that do not receive the program?" The outcome for the primary research questions is sexual intercourse in the last 3 months and the domain is sexual activity.

There are 13 secondary research questions:

1. What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of engaging in sexual intercourse in the past 3 months, 3 months after the pre-survey compared to those that do not receive the program? [domain: sexual activity, outcome: sexual intercourse in the past 3 months]
2. What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of engaging in oral sex in the past 3 months compared to those that do not receive the program? [domain: sexual activity, outcome: oral sex in the past 3 months]
3. What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports that they will make sure a condom is used if they have sex tomorrow compared to those that do not receive the program? [domain: sexual risk avoidance, outcome: intentions to consistently use condoms]
4. What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of having unprotected sex in the last three months compared to those that do not receive the program? [domain: sexual risk avoidance, outcome: frequency of unprotected sexual intercourse in the past three months]
5. What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of engaging in sexually intimate activities in the last month compared to those that did not receive the program? [domain: sexual activity, outcome: frequency of engaging in intimate activities in the past month] (6a) What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of delaying their intentions to engage in sexual intercourse compared to those that did not receive the program? [domain: sexual intentions, outcome: intent to engage in sexual intercourse] (6b) What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of delaying their intentions to engage in sexually intimate activities compared to those that did not receive the program? [domain: sexual intentions, outcome: intent to engage in sexually intimate activities] (7) What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of their confidence in their ability to resolve conflict compared to those that did not receive the program? [domain: conflict resolution skills, outcome: confidence in one's ability to resolve conflict] (8) What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of making good dating choices compared to those that did not receive the program? [domain: healthy relationships, outcome: making good dating choices] (9) What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports on their awareness of relationship safety compared to those that did not receive the program? [domain: healthy relationships, outcome: awareness of relationship safety] (10) What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' knowledge of HIV/STDs/STIs and pregnancy prevention compared to those that do not receive the program? [domain: sexual health knowledge, outcome: knowledge of HIV/STDs/STIs and pregnancy prevention] (11) What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of their confidence in their ability to negotiate abstinence and resist negative pressure to have sex compared to those that did not receive the program? [domain: sexual risk avoidance skills, outcome: confidence in ability to negotiate abstinence and resist negative pressure to have sex] (12) What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of their ability to set and meet their future goals compared to those that did not receive the program? [domain: life skills, outcome: future-oriented goals] (13) What is the effect of the 2-week YYC…MSC! program on 9th grade students in Northeast urban school districts' reports of financial literacy compared to those that do not receive the program? [domain: life skills, outcome: financial literacy].

Treatment and Comparison Conditions

Treatment Condition

The intervention/treatment participants in the treatment condition will be exposed to NJPAG's YYC…MSC! program. The program is designed to be delivered five days a week for two weeks and contains 11 lessons.

Control/Comparison Condition

The control schools were randomly selected out of a list of eligible schools (a total of 12 in the Newark Public School system). Control schools will receive their health class as "business-as-usual" with no additional programming. There is no other place that provides the YYC…MSC! program in which participants could receive programming or information. Also, there are no future plans to provide the control group with the intervention. To our knowledge, there is no other similar programming operating in the Newark Public School area.

Sample Identification, Selection, and Retention

The YYC… MSC! program is approved as the Newark Board of Education sexual education curriculum. All 9th graders in the 12 selected NPS schools will be asked to participate in the evaluation. Newark public schools with a high percentage of English Language Learners (ELL) students were eliminated since the program is only provided in English. Participants will be identified by being in a school that is randomly assigned to be a treatment school. Ninth graders in participating NPS treatment schools will receive the YYC…MSC! program.

Program attrition is predicted to be minimal, given it is a school-based program. However, attrition is likely for students who move out of the district or to another school within the district. To try to reach students that move, we are collecting contact information (phone number and personal email address) on our enrollment form. Cluster-level attrition is predicted to be unlikely given the rapport NJPAG has previously built with the Newark Public Schools individually and as a district.

Anticipated Sample Size

We currently have 12 public schools that are engaged in this study, five that will be randomly assigned to the treatment group and five that will be randomly assigned to the control group. On average, there are four health classes in each school, each of which has approximately 25 students. Therefore, each year we will enroll approximately 1,200 students in the study (25 x 4 x 12 = 1,200); 700 in the treatment group and 500 in the control group. Unless we are able to recruit more schools (we are currently having conversations with several charter schools in the Newark School District), we anticipate having to run the study for two full years, in order to meet the sample size requirements needed to achieve adequate power. We will then have 2,400 students enrolled in the study (1,400 in the treatment group and 1,000 in the control group).

Identification and selection of clusters (if applicable)

The 12 eligible Newark Public Schools were selected as clusters. There is only one exclusionary criterion for schools, and that is a high percentage of ELL students because the program is only offered in English. Given the buy-in from the Newark Board of Education, recruiting schools is not anticipated to be a problem. NJPAG has started building rapport with the department chairs at each of the participating schools. The level of randomization is at the school-level, meaning we will randomly assign each of the 12 schools to either treatment or control. Random assignment will be conducted using a random number generator from 0-1 in Excel. After generating random numbers, the 12 schools will be ordered based on their number assignment and the five with the smallest number will be assigned control and the five with the highest number will be assigned treatment. All students included in the impact analysis will be students of the participating schools prior to random assignment. Students will be notified of their schools' random assignment after the parental written notice is distributed and enrollment is completed.

Identification and selection of individuals

Youth served and recruited will be limited to 9th graders from health classrooms at the NPS sites participating in this program who do not have severe cognitive, mental, or behavioral impairments. Students with an Individual Education Plan (IEP) who are in a restricted placement due to such issues will not be able to participate in the program as they will not be a part of the mainstream classrooms where the program will take place. Students with an IEP who are placed in regular classroom settings (i.e., mainstreamed) will be included in the program.

New Jersey law only requires passive parental consent to be obtained, using a written notice about the evaluation provided to students to bring home with an option for parents to sign the notice to opt their child/student out of the evaluation. If parents wish to opt their child/student out of the evaluation, the child will still receive the program because it is the Newark Board of Education-approved sexual education curriculum treatment group health classes. When the Educator goes back to the classroom to pre-survey, they will collect any 'opt-out' notices and survey only those students whose parents have not elected to 'opt-out' their child.

Ineligible students include students with IEP who are in a restricted placement and students that bring back 'opt-out' written notices. Since we are using the same form for treatment and control schools (making it unclear to parents what group their child is in), we do not expect differences in opt-out rates between treatment and control schools.

Given NJPAG's relationship with the Newark Board of Education, recruitment of schools is not expected to be challenging. NJPAG will contact each of the eligible schools to communicate about the program, program delivery, and the timeline. Schools will be randomized and maintain their randomization category for the entirety of the evaluation for pragmatic reasons (e.g., NJPAG-school communication, building rapport, etc.). After randomization, NJPAG Educators will go into schools to tell the students about the evaluation. Students at this time will not know if they are receiving the program. Following that, students are told to bring home the written notice and provide it to their parents for the option to opt out of the evaluation. For each school, the timeline goes as follows: Educator goes into school and health classes to disseminate the written notice for youth to bring home to their parents, the Educator comes back 2 weeks later to pre-survey all participating youth, the next school day the Educator comes back to start the program and consistently teaches a lesson for the next 11 school days, the next school day a different Educator goes into the school health classes to collect post-survey data, 3-months after the pre-survey date a different Educator goes into the school health classes to collect follow-up data, and 12-months after the pre-survey date a different Educator goes into the school health classes to collect follow-up data.

For the current study, no program enrollment is necessary because the NPS Board of Education approved the YYC…MSC! program as a designated sexual health curriculum for 9th graders. Therefore, all students in NPS treatment sites will be automatically enrolled in the program; but parents can opt their child out of the evaluation. Evaluation enrollment will be obtained via parental opt-out forms.

The process for each cohort is the following: NJPAG Educator goes into school, tells youth about the program and evaluation, and disseminates parental written notices.

1. Educators collect any 'opt-out' notices that were returned.
2. All participating students take the pre-survey.
3. Students receive full YYC…MSC! Programming.
4. All participating students take the post-survey. Follow-Up
5. Participating students take a follow-up survey 3-months after the pre-survey date.
6. Participating students take a follow-up survey 12 months after the pre-survey date.

Tracking and Retention of Individuals

Because the YYC…MSC! program was approved by the Newark Board of Education as a sex education program for NPS 9th graders required health classes; program dosage and attendance will be consistent with students' health class attendance. For follow-up data collection, NJPAG Educators will go back into the school building and pull participating students from their classes to participate in follow-up surveys. If a given participant is not there when the NJPAG Educator goes back to the school to collect follow-up data, we will communicate with the school district contact to attempt to locate the participating students to survey them at their new school. If a participating student moves out of the Newark school district, we will contact them via phone or email and ask them to participate in the survey. The anticipated response rate for each round of data collection is as follows: pre-survey (95%), post-survey (85%), three months (80%), and 12 months (70%).

Data Collection

Data Collection Plan

Participants in the treatment group will be determined to have completed their engagement in the intervention if they received 80% of the program lessons (e.g., 9 out of 11). The data collection window for each time point is two weeks. The data collection window will be managed by reviewing participants' timestamp (e.g., the time and date they started the survey) on their survey completion. There are no costs associated with measures because they were all created for this project specifically.

The timing of data collection for the pre-survey is before the first lesson is administered, approximately 2 weeks after disbursing the written notice for students to bring home to their parents. The post-survey is collected immediately following the completion of the program. The 3-month follow-up survey is collected 3-months from the pre-survey date. The 12-month follow-up is collected 12-months from pre-survey.

Data Security \& Privacy

Survey data collection will be captured electronically via Alchemer.com, ensuring the quality and security of both data collection and data storage. Individual-level federal reporting and tracking data is captured and stored on AMTC \& Associates Online Participant Tracking System (OPTS) (further security information can be found below). Data collectors are trained in the protection of human subjects and best practices for successful data collection.

Electronic surveys will be collected using online surveys from Alchemer.com. Alchemer is committed to ensuring the quality and security of its data collection software and has robust monitoring software used to monitor performance and notify any issues.

All staff working with Personally Identifiable Information (PII) and sensitive data complete training on the program policy and sign a confidentiality form agreeing not to share participant-level PII or sensitive data. Access to data will only be granted to project staff who need access and sign a confidentiality agreement such as staff responsible for data collection, entry and/or electronic or hard copy transmission; these staff must be trained on and sign the receipt of this policy and confidentiality agreement. Those who may only need limited access will be trained and will sign the receipt of the policy. Limited access to data may include only having access to the necessary data to understand the number of responses for each school, class, and teacher. For example, facilitators may collect attendance data but may not need access to survey data, and yet attendance data may be related to sensitive data. Additionally, identifiable data from surveys will only be available to the analysts tasked with de-identifying the data. Documents containing PII (e.g., completed parent consent forms) will be saved on a secure computer server away from survey data. Data will be de-identified. To link entry, exit, and follow-up surveys, a pseudo-ID will be created by asking the youth for their first and last initials, birth month, and birth date on each survey administered. These individual survey items will be concatenated to create a unique ID for each respondent.

If a data breach occurs, Evaluators will notify NJPAG, pause data collection, identify any vulnerable parties, fix any issues resulting from a breach, and ensure any improperly shared information is deleted.

Analysis Plan

All analyses will be conducted using R. Outliers will be identified as three standard deviations above the mean and removed from the data. When appropriate, missing data will be deleted casewise. Participant data will be utilized for analyses if they completed 75% of scale items within a given scale. We will use a 2-level HLM model to analyze all outcomes. Level 1 covariates include baseline score, ethnicity, gender, and age. There is one Level 2 covariate, school size.