Designing a Mobile App to Support Academic Success for Student Veterans

The goals of this phase are to: 1) gain feedback on the VetEd mobile app via Veteran usability testing; 2) assess Veteran interest, satisfaction, and usability of VetEd; 3) iteratively address any technological concerns, and 4) collect preliminary information on the impact of the VetEd app on academic functioning (retention, perceived course task ability, and educational self-efficacy) and community reintegration. The investigators will conduct pilot usability testing with 15 Veterans who meet eligibility criteria, conducted in three waves. Each usability testing wave will include 5 participants, based on the finding that approximately 80% of usability problems can be detected using a sample of 4 to 5 participant per wave.

Recruitment. Participants will be 15 Veterans recruited from the Bedford VA medical center (VAMC), and local colleges. As noted in Phase 1, based on the rate of student Veterans seeking educational support services at the Bedford facility and the number of Veteran students in the community, the investigators should have no difficulty recruiting the proposed number of participants. Additionally, this research group has had no problem meeting prior, higher recruitment goals for the recent VITAL Supported Education (SEd) using similar recruitment strategies (n = 43).

Screening. Potential participants will be pre-screened by phone to determine if basic study eligibility criteria are met. Individuals who remain eligible after the pre-screen will be invited to attend a comprehensive in-person baseline session. At the beginning of the baseline session, the investigators will provide participants with detailed information about the study and obtain written informed consent. Following informed consent, participants will complete study assessments and, with staff assistance, and download the app to their personal device.

Usability testing procedures. Participants will be provided with the app and log-in information, to utilize at home over a four-week period. During that time, they will complete at least two in-app tasks within per week, including setting up an educational roadmap, tracking course assignments, learning about successful study strategies, completing a coping exercise, and finding information related to GI Bill benefits.

Participants will be evaluated after app use in the following areas: (a) app usefulness: whether users can successfully complete designated tasks on the app, (b) effectiveness: the ability to accomplish assigned educational app tasks quickly and easily, (c) learnability: the ability to meet pre-determined app navigation goals (e.g. accessing and inputting information into the education roadmap), (d) satisfaction: users' feelings and opinions about the app related to their educational functioning and alignment with mental-health needs, and (e) engagement: initial and sustained interaction between a user and a mobile application. Usability measures will include two quantitative system usability scales (SUS; AES), task completion (quantitative), app log-in and use information (engagement). Perceived usefulness, engagement, and overall satisfaction will also be assessed through a post-app-testing qualitative interview (1.5 hours).

After each round of testing, the investigative team will work with the programming team to improve the app. The following measures will be utilized: Mental Health Diagnostic Measure: The Structured Clinical Interview (SCID-5) is a reliable semi-structured instrument that is the gold standard for diagnosing mental health disorders.

System Usability Scale (SUS): The SUS a 10-item measure, scored on a 5-point Likert scale, that assesses human-computer interaction. A SUS score above 58 is regarded as above average, and a SUS score above 80 a score above which participants are likely to recommend the product to friends.

Acceptability E-Scale (AES): The AES is a 5-item validated measure of the acceptability of mobile interventions, which has a cut-off of 80% for acceptability covering overall and specific component satisfaction.

Academic Functioning Outcomes: Increased academic function is the primary non-usability related goal of the VetEd app. Consequently, preliminary educational functioning information related to academic self-efficacy, preparedness for course tasks, and perceived confidence in course completion/retention will also be collected and described. In line with educational research suggesting non-grade factors can be stronger predictors of educational success and graduation, the investigators will specifically assess academic self-efficacy (Academic Self-Efficacy Scale: ASES), ability to complete course tasks (The Coping With Education Barriers Subscale: CWBS), and confidence in completing coursework and academic retention (qualitative interview).

Community Reintegration: The Military to Civilian questionnaire (M2C-Q) is a 16-item intended to measure dimensions of community reintegration related to productivity (in education, work, and domestic life); social relations; community engagement; and perceived meaning in life, self-care, and leisure.

Data management. VetEd will be developed and hosted on a secure server by Geisel Software, Inc. The app will collect usability and content data, but will not collect personally identifiable information. Coded usernames and passwords will be used for app log-in. When data collection is complete, Geisel Software, Inc. will transfer the coded usability data to the Bedford VAMC research staff. Data will not be shared outside the VA. Paper research files will be kept at the Bedford VAMC in a locked cabinet and use deidentified numbers. Data will be checked for inconsistencies, omissions, and errors. The research assistant will be trained in data management and confidentiality procedures, and be responsible for data entry and management.

Data analysis plan. The main Phase 2 quantitative measures of acceptability and usability will be the Acceptability E-Scale and the System Usability Scale (SUS), respectively. The investigators expect that Acceptability E-Scale and SUS ratings will increase across the 3 waves of app revision, and that by the third wave the investigators will reach the benchmark of 80% on the Acceptability E-Scale and a 68 on the SUS (good acceptability). Additional data will be provided by the qualitative interviews, using rapid qualitative processes. The investigators will also assess feasibility and engagement by measuring the proportion of individuals who successfully complete the 4 week app testing, with at least 5 completed activities (of requested 8, 2 per week) or 63% of the program completed, in line with the mean number of completed sessions found in previous mobile interventions for chronic conditions. The investigators will also assess academic functioning (academic self-efficacy, course task completion, and perceived retention/course completion) and community reintegration (M2C-Q) pre- and post-app use for each student Veteran specifically to 1) characterize the sample in terms of baseline academic functioning and community reintegration, 2) describe changes in academic functioning and community reintegration, and 3) test the feasibility of these measures in anticipation of a larger, future trial.