Perception of Health Promotion Messages

Study hypotheses and design were pre-registered on OSF (https://osf.io/nzdt2/) and ethics approval for the study was obtained from The University of Queensland (2024/HE000318).

Six hundred and fourteen participants aged 16-25 years old, living in Australia were recruited through the Nielsen Consumer LLC and marketing databases provided through NielsenIQ Datasets. A quota was applied to ensure that there were equal distributions across age and gender. Informed consent was obtained from all participants prior to the start of the survey.

AI-generated, co-designed ads The detailed methodology of the co-design development process is available to as an OSF Preprint. There were 2 phases for ads generation. In Phase 1, the investigators conducted two focus group (N = 10 total participants, mean age 16.3 years) to evaluate an initial set of 100 AI-generated ads created using a basic approach of zero-shot prompting (model is given a task without any examples of existing vaping prevention), single-prompt usage (one prompt is used to interact with the model) and automatic text-image integration (automatic overlaying of AI-generated text onto corresponding AI-generated images).

Based on the feedback from Phase 1, the investigators refined the approach for Phase 2 to a few-shot prompting approach (model is given a task along with a few examples in the prompt to guide its response) with multi-prompt usage (multiple sequential prompts are used to refine or build upon previous interactions with the model until ads meet quality criteria for accuracy, relevance, and persuasion attempt) and manual text-image integration (text and images) by TS to ensure strong text-image cohesion. Twenty-five AI-generated ads were created using Claude 3 for text and Midjourney v6 for images. These refined ads were evaluated through nine semi-structured interviews (N = 9, mean age 16.3 years).

The feedback from Phase 2 interviews was used to further refine the ads in Phase 3, resulting in a final set of 25 ads, based on five themes: addiction, financial impact, health consequences, industry manipulation and social norms.

Existing ads from official health agencies The investigators systematically identified vaping prevention ads from official health agencies using a two-phase search. First, official health agencies from the U.S. with active anti-vaping campaigns, such as the World Health Organization (WHO), U.S. Food and Drug Administration (FDA), Centers for Disease Control and Prevention (CDC), and National Health Service (NHS, UK) were searched and identified. Second, a comprehensive searches of the health agencies' official websites and social media accounts (Facebook, Instagram, Twitter) using keywords such as "vaping prevention," "e-cigarette awareness," and "youth vaping" were conducted.

To ensure consistency and quality, strict inclusion criteria for the ads were established. Ads were included if they: (1) were created by an official health agency, (2) related to one of our five pre-determined themes, (3) were youth-oriented, (4) were in English, and (5) were available as picture files that were in square format or could be cropped to square format.

Twenty-five ads, prioritising those from WHO and FDA due to their global reach and extensive youth-focused campaigns, were selected. The logos and other descriptors of the health agency that created them were carefully removed. All 50 ads are publicly available through our GitHub repository: https://github.com/gckc123/AIvaping.

Procedure This study employed a 2 (material source: AI-generated vs. existing health agency) by 4 (source labelling conditions: control condition, AI condition, health agency condition and combined condition) mixed experimental design, where ad source was manipulated within-subjects and source labelling was between subjects. Each participant viewed and evaluated 50 ads in random order, of which 25 were generated by AI in the co-design process, and 25 were existing ads from official health agencies, giving a total of 30,700 observations.

Participants were randomly assigned to one of four source labelling conditions to examine how labelling influenced ad evaluation: (1) Control condition where ads were presented without any source labelling, (2) AI condition where ads were labelled with the text "Made with AI", (3) Health agency condition where ads were labelled with the text "Made by the World Health Organization (WHO)", and (4) Combined condition where ads were labelled with the text "Made with AI, by the World Health Organization (WHO)".

Measures Perceived Message Effectiveness (PME) Each ad were rated with five items adapted from the validated University of North Carolina (UNC) Perceived Message Effectiveness (PME) Scale for Youth. Two items assessed effects perceptions, i.e. the ads' potential to change attitudes and behaviours, and three items assessed ad perceptions, i.e., judgements about ad characteristics.

The effects perceptions included the following two survey items: "This ad makes me think vaping is _____" (1 = a very bad idea, 4 = neither a good nor a bad idea, 7 = a very good idea) and "This ad _____ from vaping" (from 1 = strongly discourages me to 7 = Strongly encourages me). Lower score indicates stronger effect perceptions. The ad perceptions included the following three survey items: "This ad grabbed my attention.", "This ad provided useful information." and "This ad was convincing." Participants rated these items on a 7-point Likert scale from 1 (Strongly disagree) to 7 (Strongly agree). Higher score indicates stronger ad perception.

2.4.4 Manipulation Check At the end of the experiment, participants were asked "Throughout this study, you viewed several health ads about vaping. How were these ads labelled?" Participants then selected the response option that best matched what they remembered seeing including "The ads were labelled as "made with AI", "The ads were labelled as "made with AI, by the World Health Organization (WHO)", "The ads were labelled as "made by the World Health Organization (WHO)", "The ads did not have any specific label".

Data analysis All analyses were conducted in R version 4.3.3 with the lmer and lmerTest packages. Linear mixed models were used to test the difference in rating between the AI-generated ads and existing ads from official health agencies. For each of the five PME measures, two models were fitted, one with the labelling conditions and one without. Likelihood ratio tests were used to test the overall effect of labelling. Two random intercepts were specified, one was used to account for the repeated measure from participants and one was used to account for the repeated presentation of ads across participants. The non-inferiority margins were set at 0.5 points in the 7-point scale.

For the hypothesis regarding labelling effects (H2), main analysis using the full sample (N = 614) and sensitivity analyses using only the 462 participants (75%) who correctly selected their assigned labelling condition in the manipulation check were conducted.