AzTrials
AzTrials

Surgical EducationNCT04700384

Effectiveness of an Artificial Intelligent Tutoring System in Simulation Training

A study testing Virtual Operative Assistant Training and Remote-Based Expert Instructor Training in people with Surgical Education, run by McGill University at 1 site in Canada.

Completed

Finished. The study has ended and is not enrolling. Results may be posted on the official record.

Official record
Search all trials

Trial summary

Brief Summary: Background: Although surgical experience and technical skill are associated with better patient outcomes, quantitating surgical ability in the operating room is challenging. Read more about this study

Who is it for?
Any ageHealthy volunteers can take part
Conditions the study is for
Surgical Education
What is being tested?
Virtual Operative Assistant Training and Remote-Based Expert Instructor Training
Could I receive a placebo?
No, everyone receives an active treatment
How many people
70 participants
Where
1 siteCanada
Study dates
15 Jan 2021to 15 May 2021
Phase
Interventional
Are there results?
None posted in the registry

Key requirements

Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.

  • Ages Any age
  • All sexes
  • Healthy volunteers accepted
  • Child, Adult, Older adult

Inclusion Criteria:• First- and second-year medical students from any Canadian institution who do not meet the exclusion criteria.

-

Exclusion Criteria: • Participation in any of our group's previous trials involving the NeuroVR (CAE Healthcare) simulator.

-

Trial locations and contacts

1 site in 1 country.

Canada1 site
  • Neurosurgical Simulation and Artificial Intelligence Learning Centre

    Montreal, Quebec, H3A 2B4

Other trials in Canada

About this study

Official title: Comparing the Virtual Operative Assistant to Expert-based Instruction in Surgical Education: A Randomized Controlled Trial

Brief Summary:

Background:

Read more

Although surgical experience and technical skill are associated with better patient outcomes, quantitating surgical ability in the operating room is challenging. In surgical education, large datasets generated by high-fidelity virtual reality simulators can be employed by machine learning algorithms to objectively measure trainee performance and competence on expert benchmarks. This allows repetitive practice of surgical skills in safe and risk-free environments with immediate feedback.

Our group developed and has a patent pending for an intelligent tutoring system called the Virtual Operative Assistant (VOA). Utilizing an Artificial Intelligence (AI) support vector machine algorithm, the VOA assesses data derived from the NeuroVR (CAE Healthcare) simulator platform and provides individualized audiovisual feedback to improve learner performance during simulated brain tumor resections. The effectiveness of intelligent tutoring systems such as the VOA to the human surgical apprenticeship pedagogy remains to be elucidated.

The aim of this study is to compare the effectiveness and educational impact of personalized VOA feedback to expert instruction on medical student's technical skills learning of a virtual reality tumor resection procedure.

Specific Aims: 1) To assess if medical students receiving personalized VOA feedback statistically improve their surgical performance when compared to those having (a) no expert instructor feedback or (b) expert instructor-mediated feedback. 2) To outline if different emotions are elicited by the VOA intelligent tutoring system in medical students while performing this achievement task as compared to human instruction

Read the detailed description

Design: A three-arm partially blinded randomized controlled trial of VOA training versus remote-based expert instruction versus control.

Setting: Neurosurgical Simulation and Artificial Intelligence Learning Centre, Montreal Neurological Institute.

Participants: Eligible first- and second-year medical students from across the province of Quebec.

Task: Complete removal of a simulated tumour - distinguishable by colour and haptic properties - with minimal bleeding and damage to surrounding healthy brain using two surgical instruments (Cavitron Ultrasonic Aspirator and Bipolar pincers) of the NeuroVR (CAE Healthcare) surgical simulator.

Intervention: A single 75-minute training session, including six virtual subpial tumour resection attempts (five simple practice scenarios and one complex realistic scenario) with assessment and feedback from either:

  1. the VOA intelligent tutoring system (Group 2) or
  2. a remote-based expert instructor (Group 3) Both compared to:
  3. control group (Group 1) that receives no assessment or performance feedback.

To our knowledge this will be the first study to compare the effectiveness of an AI-powered intelligent tutoring system to expert instruction in the context of medical and surgical virtual reality simulation and assess the emotional response to such instruction. This study aims to begin to identify successful approaches to use this innovative technology in the medical educational curriculum and improve patient outcomes by augmenting safety, efficiency and competency of surgeons and other healthcare providers.

Being tested
Virtual Operative Assistant Training, Remote-Based Expert Instructor Training
Keywords
Virtual Reality, Simulation

What participants receive

What participants receive. In a randomised study each person is assigned to one group by chance.

  • Virtual Operative Assistant TrainingBehavioural

    Individuals receive the same basic information, have the same amount of time and perform the same scenarios as the control group. In the 5-minutes between attempts, participant receive the Virtual Operative Assistant assessment of their performance and audiovisual feedback.

  • Remote-Based Expert Instructor TrainingBehavioural

    Individuals receive the same information, have the same amount of time and perform the same scenarios as the control group. Meanwhile, a trained instructor observes the participant's on-screen performance, that is live-streamed, remotely. Instructors are senior neurosurgery residents with extensive experience in performing and assessing this scenario. During the 5-minute feedback session, they chat with the student, discussing the performance and help in setting goals for the next trial.

Study groups in NCT04700384
Study groupRoleWhat they receive
Control GroupNo interventionControl Group - Baseline Training 25 Participants allocated. Individuals receive introductory information on using the simulator and the scenario. They perform 5 simple subpial tumour resections for practice and have 5 minutes per trial. After each attempt, the student takes a 5-minute break with no assessment or feedback on their performance. On their 6th attempt they have 13 minutes to perform a different realistic scenario.
Experimental Group - Virtual Operative Assistance TrainingExperimentalExperimental Group - Virtual Operative Assistance Training 25 participants allocated. Individuals receive the same information, have the same amount of time and perform the same scenarios as the control group. In the 5-minutes between attempts, participant receive the Virtual Operative Assistance Training assessment of their performance and audiovisual feedback.
Receives: Behavioral: Virtual Operative Assistant Training
Experimental Group - remote-based expert Instructor TrainingExperimental25 participants allocated. Individuals receive the same information, have the same amount of time and perform the same scenarios as the control group. Meanwhile, a trained instructor observes the participant's on-screen performance, that is live-streamed, remotely. Instructors are senior neurosurgery residents with extensive experience in performing and assessing this scenario. During the 5-minute feedback session, they chat with the student, discussing the performance and help in setting goals for the next trial.
Receives: Behavioral: Remote-Based Expert Instructor Training

What is being measured

The primary outcome is the main question the study is designed to answer.

  1. Change in procedural performance .

    Performance in each practice attempt is measured utilizing raw data from the simulator that is used for assessment by previously established AI algorithms on validated metrics.

    Measured: Day of Study

  2. Change in learning

    Performance on the complex realistic scenario is evaluated by expert instructors using the Objective Structured Assessments of technical Skills (OSATS) Visual Rating Scale (weighted at 50%) and the AI assessment algorithms (weighted at 50%) creating a composite performance score.

    Measured: Day of Study

Secondary outcomes (2)
  1. Difference in the strength of emotions elicited

    Measured using Duffy's Medical Emotions Scale (MES), before, during and after the intervention.

    Measured: Day of Study

  2. Difference in cognitive load

    Measured using Leppink's Cognitive Load Index (CLI) after the intervention.

    Measured: Day of Study

Timeline

From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.

  1. First posted7 Jan 2021
  2. Study start15 Jan 2021
  3. Primary completion15 May 2021
  4. Study completion15 May 2021

Resources

More details

For doctors and researchers: how the study is designed, who runs it, and its registry IDs.

Study design
Study type
Interventional
Phase
Phase N/A
Primary purpose
Health services research
Allocation
Randomised
Intervention model
Parallel groups
Model description
Randomized Controlled Trial
Masking
Double blind
Who is masked
Participant, Outcomes assessor
Enrolment
70 participants (actual)
IDs, dates and publications
ClinicalTrials.gov ID
NCT04700384
Sponsor's protocol ID
2010-270, NEU-09-042
Status last verified
May 2021
First submitted
6 Jan 2021
Last updated
24 May 2021
Results posted
No
Data monitoring committee
No
FDA-regulated drug
No
FDA-regulated device
No
Individual data sharing
Yes: Data obtained from primary and secondary outcomes may be shared if other researchers have an interest in this data

Publications

  • Birkmeyer JD, Finks JF, O'Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ; Michigan Bariatric Surgery Collaborative. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013 Oct 10;369(15):1434-42. doi: 10.1056/NEJMsa1300625.

    PubMed 24106936
  • Birkmeyer JD, Stukel TA, Siewers AE, Goodney PP, Wennberg DE, Lucas FL. Surgeon volume and operative mortality in the United States. N Engl J Med. 2003 Nov 27;349(22):2117-27. doi: 10.1056/NEJMsa035205.

    PubMed 14645640
  • Stulberg JJ, Huang R, Kreutzer L, Ban K, Champagne BJ, Steele SR, Johnson JK, Holl JL, Greenberg CC, Bilimoria KY. Association Between Surgeon Technical Skills and Patient Outcomes. JAMA Surg. 2020 Oct 1;155(10):960-968. doi: 10.1001/jamasurg.2020.3007.

    PubMed 32838425
  • Winkler-Schwartz A, Yilmaz R, Mirchi N, Bissonnette V, Ledwos N, Siyar S, Azarnoush H, Karlik B, Del Maestro R. Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. JAMA Netw Open. 2019 Aug 2;2(8):e198363. doi: 10.1001/jamanetworkopen.2019.8363.

    PubMed 31373651
  • Mirchi N, Bissonnette V, Yilmaz R, Ledwos N, Winkler-Schwartz A, Del Maestro RF. The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine. PLoS One. 2020 Feb 27;15(2):e0229596. doi: 10.1371/journal.pone.0229596. eCollection 2020.

    PubMed 32106247
  • Winkler-Schwartz A, Marwa I, Bajunaid K, Mullah M, Alotaibi FE, Bugdadi A, Sawaya R, Sabbagh AJ, Del Maestro R. A Comparison of Visual Rating Scales and Simulated Virtual Reality Metrics in Neurosurgical Training: A Generalizability Theory Study. World Neurosurg. 2019 Jul;127:e230-e235. doi: 10.1016/j.wneu.2019.03.059. Epub 2019 Mar 15.

    PubMed 30880209
  • Duffy, M.C., et al., Emotions in medical education: Examining the validity of the Medical Emotion Scale (MES) across authentic medical learning environments. Learning and Instruction, 2020. 70: p. 101150.

  • Leppink J, Paas F, Van der Vleuten CP, Van Gog T, Van Merrienboer JJ. Development of an instrument for measuring different types of cognitive load. Behav Res Methods. 2013 Dec;45(4):1058-72. doi: 10.3758/s13428-013-0334-1.

    PubMed 23572251
  • Fazlollahi AM, Bakhaidar M, Alsayegh A, Yilmaz R, Winkler-Schwartz A, Mirchi N, Langleben I, Ledwos N, Sabbagh AJ, Bajunaid K, Harley JM, Del Maestro RF. Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial. JAMA Netw Open. 2022 Feb 1;5(2):e2149008. doi: 10.1001/jamanetworkopen.2021.49008.

    PubMed 35191972

Questions about this trial

What is this clinical trial studying?

A study testing Virtual Operative Assistant Training and Remote-Based Expert Instructor Training in people with Surgical Education, run by McGill University at 1 site in Canada.

The full registered description is in the About this study section and the official record.

Is this clinical trial currently recruiting?

The record says: finished, as of the sponsor's last update on 24 May 2021. Status can change between updates, so confirm with the study contact before making plans.

Where is the study taking place?

The record lists 1 site in Canada. Whether a site is taking part right now should be confirmed with that site. See every site.

What are the possible benefits and risks?

They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.

Are results available for this study?

The study is completed, but results are not available in the source record. That does not mean the treatment did or did not work.

How can a doctor refer a patient or contact the research team?

The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.

What does the primary outcome measure?

Change in procedural performance . (time frame: Day of Study). The study lists 2 primary outcomes in all. It is what the study was designed to measure, not a finding.

Where can I read the original study information?

On the ClinicalTrials.gov record for NCT04700384, which also lists 9 linked publications.

Does a listing mean the treatment is approved or proven to work?

No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.

How often is this information updated?

AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 24 May 2021.

Free account

Log in to keep searching

You’ve used your 3 free searches. AzTrials is free: log in or create a free account to keep searching clinical trials and save the ones you want to follow.

Create a free accountLog in