Educational ActivitiesAIHSSNCT05963802
Evaluation of the Efficacy and Usability of Artificial Intelligence (ChatGPT) for Health Sciences Students
A study testing Artificial Intelligence (ChatGPT) Group in people with Educational Activities and AI (Artificial Intelligence), run by Carleton University at 1 site in Canada.
Trial summary
Crossover Randomized Control trial, in which subjects are randomly assigned to one of two groups: one (ChatGPT) receiving the intervention that is being tested, and the control group receiving usual online resources. Read more about this study
- Who is it for?
- Ages 18 years and olderHealthy volunteers can take part
- Conditions the study is for
- Educational Activities, AI (Artificial Intelligence)
- What is being tested?
- Artificial Intelligence (ChatGPT) Group
- Could I receive a placebo?
- No, everyone receives an active treatment
- How many people
- 26 participants
- Where
- 1 siteCanada
- Study dates
- 1 Sep 2023to 1 Dec 2023
- Phase
- Interventional
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages 18 years and older
- All sexes
- Healthy volunteers accepted
- Adult, Older adult
Who may be able to join
- Health Science students
- Must be registered for the course Chronic Health Conditions and Disability (HLTH 3503), Fall 2023.
Who cannot join
- Unable to provide informed consent
Trial locations and contacts
1 site in 1 country.
About this study
Official title: Crossover Randomized Controlled Trial to Evaluate the Efficacy and Usability of Artificial Intelligence (ChatGPT) for Health Sciences Students (AIHSS)
Crossover Randomized Control trial, in which subjects are randomly assigned to one of two groups: one (ChatGPT) receiving the intervention that is being tested, and the control group receiving usual online resources.
Read the detailed description
A two-stage randomization strategy will be adopted. The allocation of participants into event groups A and B will be determined through computer-generated randomization in an allocation ratio: 1:1. To address ethical concerns regarding power dynamics between professors and students; the investigators have opted to involve an individual who is independent of the research team. This decision aims to minimize any potential bias or influence that could arise from direct professor involvement in the intervention. This approach helps ensure that the study is conducted fairly and unbiasedly, fostering a research environment that prioritizes equal opportunities for all participants. Both groups will simultaneously work on an assignment (A and B).
The investigators employed a cross-over design to ensure equity in the learning experience for all students. This involved assigning students either the ChatGPD (A) or the regular online tools (B) for accessing resources and assisting them with their assignments. During the initial phase of the intervention, participants in group A will utilize ChatGPT to complete their assignments, while participants in Group B will serve as the control group. In the subsequent period, the interventions will switch between the two groups. To assess the participants' technology usability, both groups will complete pre-and post-questionnaires. Additionally, participants will be requested to fill out a post-intervention questionnaire to evaluate their perceptions of the assistance methods. After a period of a wash-out of 13 days/week, participants who were previously allocated to Group A will be allocated to Group B and the same with participants previously allocated to Group B will be to Group A. This crossover randomized control trial will include 50 health sciences students who are registered to the course Chronic Health Conditions and Disability ( HLTH 3503), Carleton University. During the first part of the intervention, A group will use ChatGPT to complete their assignment, while B participants will act as a control group. In the second week, the interventions will cross over. Both groups' participants will complete a pre-and post-questionnaire to assess their knowledge. Participants will also complete a post-intervention questionnaire to evaluate their perceptions. The System Usability Scale (SUS) will be administered at the end of the interventions and when participants cross over to the other assistance tool. This allows for the assessment of outcomes and comparisons within and between the two intervention arms. By collecting data at multiple points in the study, the investigators can evaluate the effects of each intervention and any potential carryover effects during the crossover period. Additionally, it provides an opportunity to measure changes in outcomes within individual participants as they switch from one intervention to another. The questionnaire about socioeconomic, literacy questions, Benefits of AI and risks and negative experiences associated with AI usage.
- Being tested
- Artificial Intelligence (ChatGPT) Group
- Keywords
- Artificial intelligence, Education, Health Science Students
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
Artificial Intelligence (ChatGPT) GroupOther
Participants in Group A will be assigned to utilize ChatGPT as their tool to complete assignments. They will be given a period of six days to utilize artificial intelligence through ChatGPT for assignment completion. Along with the assignment instructions, participants will receive an ethical guideline and specific guidelines on how to effectively utilize ChatGPT. Once the intervention period is concluded, participants will be given a 24-hour window to complete a survey that assesses the usability of the technology. The survey aims to gather valuable feedback on the participants' experience and perception of using ChatGPT for their assignments. In addition, participants in Arm 1 will also be asked to fill out a survey regarding their perception of using artificial intelligence (AI) as an assistance tool to complete their assignments. This survey aims to gather insights into their thoughts, opinions, and attitudes towards utilizing AI in the learning process.
| Study group | Role | What they receive |
|---|---|---|
| Traditional Online tools Group | No intervention | Group B will serve as the control group in this study. Participants in this group will receive instructions on how to complete the assignment using traditional online tools available on the internet, without the use of artificial intelligence. They will have a timeframe of six days to complete the assignment using these conventional tools. Similar to participants in Arm 1, they will also be required to fill out a survey on technology usability, providing feedback on their experience with the online tools. |
| Artificial Intelligence (ChatGPT) Group | Experimental | Participants in Group A will be assigned to utilize ChatGPT as their tool to complete assignments. They will be given a period of six days to utilize artificial intelligence through ChatGPT for assignment completion. Along with the assignment instructions, participants will receive an ethical guideline and specific guidelines on how to effectively utilize ChatGPT. Once the intervention period is concluded, participants will be given a 24-hour window to complete a survey that assesses the usability of the technology. The survey aims to gather valuable feedback on the participants' experience and perception of using ChatGPT for their assignments. In addition, participants in Arm 1 will also be asked to fill out a survey regarding their perception of using artificial intelligence (AI) as an assistance tool to complete their assignments. This survey aims to gather insights into their thoughts, opinions, and attitudes towards utilizing AI in the learning process Receives: Other: Artificial Intelligence (ChatGPT) Group |
What is being measured
The primary outcome is the main question the study is designed to answer.
System Usability Scale (SUS) scores are 0-100,The average SUS score is 68.
Technology usability
Measured: within 24 hours
Secondary outcomes (1)
Student's perception of artificial intelligence, questionnaire reviwed by experts, No recognised gold standard measurement for outcome.
Student's perception about the use of artificial intelligence
Measured: within 24 hours
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted27 Jul 2023
- Study start1 Sep 2023
- Primary completion1 Dec 2023
- Study completion1 Dec 2023
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Publications (5)Papers the sponsor has linked to this study.
- Other trials for Educational ActivitiesStudies of the same condition, open and under way.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Other
- Allocation
- Randomised
- Intervention model
- Crossover
- Model description
- We employed a crossover design to ensure equity in the learning experience for all students. This involved assigning students either the ChatGPD (A) or the regular online tools (B) for accessing resources and assisting them with their assignments. During the initial phase of the intervention, participants in group A will utilize ChatGPT to complete their assignments, while participants in Group B will serve as the control group. In the subsequent period, the interventions will switch between the two groups.
- Masking
- Single blind
- Who is masked
- Investigator
- Enrolment
- 26 participants (actual)
Sponsor and investigators
- Lead sponsor
- Carleton University
- Sponsor type
- University, hospital or other
- Collaborators
- Université de Montréal
- Responsible party
- Principal investigator, Mirella Veras, Carleton University
- Investigators
- Mirella Veras, PhD, Principal investigator, Carleton University
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT05963802
- Sponsor's protocol ID
- AIHSS01
- Acronym
- AIHSS
- Status last verified
- Feb 2024
- First submitted
- 4 Jul 2023
- Last updated
- 6 Feb 2024
- Results posted
- No
- Data monitoring committee
- Yes
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- No
Publications
Dwan K, Li T, Altman DG, Elbourne D. CONSORT 2010 statement: extension to randomised crossover trials. BMJ. 2019 Jul 31;366:l4378. doi: 10.1136/bmj.l4378.
PubMed 31366597Luna A, Casertano L, Timmerberg J, O'Neil M, Machowsky J, Leu CS, Lin J, Fang Z, Douglas W, Agrawal S. Artificial intelligence application versus physical therapist for squat evaluation: a randomized controlled trial. Sci Rep. 2021 Sep 13;11(1):18109. doi: 10.1038/s41598-021-97343-y.
PubMed 34518568Liu X, Rivera SC, Moher D, Calvert MJ, Denniston AK; SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI Extension. BMJ. 2020 Sep 9;370:m3164. doi: 10.1136/bmj.m3164.
PubMed 32909959Zhou L, Bao J, Setiawan IMA, Saptono A, Parmanto B. The mHealth App Usability Questionnaire (MAUQ): Development and Validation Study. JMIR Mhealth Uhealth. 2019 Apr 11;7(4):e11500. doi: 10.2196/11500.
PubMed 30973342Veras M, Dyer JO, Rooney M, Barros Silva PG, Rutherford D, Kairy D. Usability and Efficacy of Artificial Intelligence Chatbots (ChatGPT) for Health Sciences Students: Protocol for a Crossover Randomized Controlled Trial. JMIR Res Protoc. 2023 Nov 24;12:e51873. doi: 10.2196/51873.
PubMed 37999958
Study at a glance
From the registry record, last updated by the sponsor on 6 Feb 2024. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Crossover Randomized Controlled Trial to Evaluate the Efficacy and Usability of Artificial Intelligence (ChatGPT) for Health Sciences Students (AIHSS)
- ClinicalTrials.gov ID
- NCT05963802
- Acronym / sponsor ID
- AIHSS / AIHSS01
- Keywords
- Artificial intelligence, Education, Health Science Students
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 26 participants (actual)
- Lead sponsor
- Carleton UniversityUniversity, hospital or other
- Collaborators
- Université de Montréal
- Study start
- 1 Sep 2023 (actual)
- Primary completion
- 1 Dec 2023 (actual)
- Study completion
- 1 Dec 2023 (actual)
- First posted
- 27 Jul 2023 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
Crossover Randomized Control trial, in which subjects are randomly assigned to one of two groups: one (ChatGPT) receiving the intervention that is being tested, and the control group receiving usual online resources.
Detailed description
A two-stage randomization strategy will be adopted. The allocation of participants into event groups A and B will be determined through computer-generated randomization in an allocation ratio: 1:1. To address ethical concerns regarding power dynamics between professors and students; the investigators have opted to involve an individual who is independent of the research team. This decision aims to minimize any potential bias or influence that could arise from direct professor involvement in the intervention. This approach helps ensure that the study is conducted fairly and unbiasedly, fostering a research environment that prioritizes equal opportunities for all participants. Both groups will simultaneously work on an assignment (A and B).
The investigators employed a cross-over design to ensure equity in the learning experience for all students. This involved assigning students either the ChatGPD (A) or the regular online tools (B) for accessing resources and assisting them with their assignments. During the initial phase of the intervention, participants in group A will utilize ChatGPT to complete their assignments, while participants in Group B will serve as the control group. In the subsequent period, the interventions will switch between the two groups. To assess the participants' technology usability, both groups will complete pre-and post-questionnaires. Additionally, participants will be requested to fill out a post-intervention questionnaire to evaluate their perceptions of the assistance methods. After a period of a wash-out of 13 days/week, participants who were previously allocated to Group A will be allocated to Group B and the same with participants previously allocated to Group B will be to Group A. This crossover randomized control trial will include 50 health sciences students who are registered to the course Chronic Health Conditions and Disability ( HLTH 3503), Carleton University. During the first part of the intervention, A group will use ChatGPT to complete their assignment, while B participants will act as a control group. In the second week, the interventions will cross over. Both groups' participants will complete a pre-and post-questionnaire to assess their knowledge. Participants will also complete a post-intervention questionnaire to evaluate their perceptions. The System Usability Scale (SUS) will be administered at the end of the interventions and when participants cross over to the other assistance tool. This allows for the assessment of outcomes and comparisons within and between the two intervention arms. By collecting data at multiple points in the study, the investigators can evaluate the effects of each intervention and any potential carryover effects during the crossover period. Additionally, it provides an opportunity to measure changes in outcomes within individual participants as they switch from one intervention to another. The questionnaire about socioeconomic, literacy questions, Benefits of AI and risks and negative experiences associated with AI usage.
Study design
- Primary purpose
- Other
- Allocation
- Randomised
- Intervention model
- Crossover
- Masking
- Single blind (investigator)
- Number of arms
- 2
- Model description
- We employed a crossover design to ensure equity in the learning experience for all students. This involved assigning students either the ChatGPD (A) or the regular online tools (B) for accessing resources and assisting them with their assignments. During the initial phase of the intervention, participants in group A will utilize ChatGPT to complete their assignments, while participants in Group B will serve as the control group. In the subsequent period, the interventions will switch between the two groups.
| Arm | Type | Receives |
|---|---|---|
| Traditional Online tools Group | No intervention | –Group B will serve as the control group in this study. Participants in this group will receive instructions on how to complete the assignment using traditional online tools available on the internet, without the use of artificial intelligence. They will have a timeframe of six days to complete the assignment using these conventional tools. Similar to participants in Arm 1, they will also be required to fill out a survey on technology usability, providing feedback on their experience with the online tools. |
| Artificial Intelligence (ChatGPT) Group | Experimental | Other: Artificial Intelligence (ChatGPT) GroupParticipants in Group A will be assigned to utilize ChatGPT as their tool to complete assignments. They will be given a period of six days to utilize artificial intelligence through ChatGPT for assignment completion. Along with the assignment instructions, participants will receive an ethical guideline and specific guidelines on how to effectively utilize ChatGPT. Once the intervention period is concluded, participants will be given a 24-hour window to complete a survey that assesses the usability of the technology. The survey aims to gather valuable feedback on the participants' experience and perception of using ChatGPT for their assignments. In addition, participants in Arm 1 will also be asked to fill out a survey regarding their perception of using artificial intelligence (AI) as an assistance tool to complete their assignments. This survey aims to gather insights into their thoughts, opinions, and attitudes towards utilizing AI in the learning process |
Who may qualify
- Ages
- 18 years and older
- Sex
- All
- Healthy volunteers
- Accepted
- Age groups
- Adult, Older adult
Key inclusion criteria (2 listed)
- Health Science students
- Must be registered for the course Chronic Health Conditions and Disability (HLTH 3503), Fall 2023.
Key exclusion criteria (1 listed)
- Unable to provide informed consent
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| Artificial Intelligence (ChatGPT) GroupParticipants in Group A will be assigned to utilize ChatGPT as their tool to complete assignments. They will be given a period of six days to utilize artificial intelligence through ChatGPT for assignment completion. Along with the assignment instructions, participants will receive an ethical guideline and specific guidelines on how to effectively utilize ChatGPT. Once the intervention period is concluded, participants will be given a 24-hour window to complete a survey that assesses the usability of the technology. The survey aims to gather valuable feedback on the participants' experience and perception of using ChatGPT for their assignments. In addition, participants in Arm 1 will also be asked to fill out a survey regarding their perception of using artificial intelligence (AI) as an assistance tool to complete their assignments. This survey aims to gather insights into their thoughts, opinions, and attitudes towards utilizing AI in the learning process. | Other | Artificial Intelligence (ChatGPT) Group |
Primary outcome
System Usability Scale (SUS) scores are 0-100,The average SUS score is 68.
Time frame: within 24 hours
Technology usability
1 secondary outcome
Student's perception of artificial intelligence, questionnaire reviwed by experts, No recognised gold standard measurement for outcome.
Time frame: within 24 hours
Results and safety
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 6 Feb 2024.
- Results publication: PubMed 31366597Dwan K, Li T, Altman DG, Elbourne D. CONSORT 2010 statement: extension to randomised crossover trials. BMJ. 2019 Jul 31;366:l4378. doi: 10.1136/bmj.l4378.
- Results publication: PubMed 34518568Luna A, Casertano L, Timmerberg J, O'Neil M, Machowsky J, Leu CS, Lin J, Fang Z, Douglas W, Agrawal S. Artificial intelligence application versus physical therapist for squat evaluation: a randomized controlled trial. Sci Rep. 2021 Sep 13;11(1):18109. doi: 10.1038/s41598-021-97343-y.
- Results publication: PubMed 32909959Liu X, Rivera SC, Moher D, Calvert MJ, Denniston AK; SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI Extension. BMJ. 2020 Sep 9;370:m3164. doi: 10.1136/bmj.m3164.
- Results publication: PubMed 30973342Zhou L, Bao J, Setiawan IMA, Saptono A, Parmanto B. The mHealth App Usability Questionnaire (MAUQ): Development and Validation Study. JMIR Mhealth Uhealth. 2019 Apr 11;7(4):e11500. doi: 10.2196/11500.
- Derived from this study: PubMed 37999958Veras M, Dyer JO, Rooney M, Barros Silva PG, Rutherford D, Kairy D. Usability and Efficacy of Artificial Intelligence Chatbots (ChatGPT) for Health Sciences Students: Protocol for a Crossover Randomized Controlled Trial. JMIR Res Protoc. 2023 Nov 24;12:e51873. doi: 10.2196/51873.
- Individual participant dataSharing planned: No
Related trials for Educational Activities
Active, not recruiting
MOVI-OLE! [Open Learning Environments]
Sedentary Behavior, Physical Activity
Recruiting
Brain Research Apprenticeships in New York at Columbia
Educational Activities
Recruiting
Clinical Learning Through Extended Immersion in Medical Simulation (CLEIMS)-A New Approach to Teaching Clinical Ethics
Educational Activities, Patient Simulation
Active, not recruiting
Feasibility Study of the Pilot Post-primary Active School Flag (PPASF) Program
Physical Activity, Health Behavior
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
Mirella Veras, PhD principal investigator
Carleton University
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing Artificial Intelligence (ChatGPT) Group in people with Educational Activities and AI (Artificial Intelligence), run by Carleton University at 1 site in Canada.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 2 inclusion and 1 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 6 Feb 2024. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in Canada. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
System Usability Scale (SUS) scores are 0-100,The average SUS score is 68. (time frame: within 24 hours). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT05963802, which also lists 5 linked publications.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 6 Feb 2024; AzTrials last refreshed its copy on 7 Nov 2026.