Proficiency Based Training in Surgery - Employing An Arthroscopic Bankart Repair

Background: Simulation-based skills training has been proposed as a method of augmenting traditional in vivo training of surgeons.[1] Simulation based training is still in its infancy and there is considerable variability in the United States and around the world on how this type of training is configured.[2] The first successful simulation validation study used a proficiency-based progression methodology.[3] Trainees were required to demonstrate a performance benchmark (based on experienced surgeons performance on a simulator) before completing a laparoscopic cholecystectomy in vivo.[4] This approach has not been used in arthroscopy. In this study we propose to evaluate the utility of proficiency based skills training for learning how to perform an arthroscopic Bankart repair for unidirectional anterior shoulder instability.

Proficiency will be determined using metric based performance levels of surgeons very experienced in performing an arthroscopic Bankart procedure.[5] Metrics have previously been identified, operationally defined and agreed upon at an Arthroscopy Association of North America Delphi panel meeting and then validated. Metrics which have demonstrated construct validity will be used to define a performance benchmark/level of proficiency. Group C will be required to train on a simulator model until they demonstrate this predetermined performance level. At the completion of training all trainees are required to complete a Bankart procedure on a fresh cadaver shoulder specimen. Operative performance will be video recorded and subsequently assessed by two experienced arthroscopic surgeons blinded to subject identity and group. Raters will have been trained to reliably identify predefined procedural steps, errors and sentinel errors to an inter-rater reliability level of r >0.8. [6]

Hypothesis: It is hypothesized that proficiency based progression trainees will perform an arthroscopic Bankart procedure significantly better on a fresh cadaver shoulder than traditionally trained subjects. They will also perform significantly better than trainees who received the exact same curricular material,including the use of a simulator, but will be allowed to progress at their own desired pace with no requirement to demonstrate performance of skills to a proficiency benchmark before progressing in training.

Procedure: Three groups of orthopaedic surgical trainees in postgraduate years four and five will be provided online access to orientation videos demonstrating how to perform a standard arthroscopic Bankart repair. A set of baseline assessments of all 3 groups will be obtained prior to the beginning of training and include visuospatial,[7] perceptual,[8,9] and psychomotor evaluations[10-13] to ensure that the groups are comparable. The standard training group (Group A) will then progress with the training as they would today during a routine resident course. Groups B and C will be prospectively randomized (using an online randomizer) to their respective training conditions. Group C will study the online didactic information until they have demonstrated a predefined proficiency criterion. In addition, successful completion and proficiency demonstration with arthroscopic knot tying will be a requirement for them to proceed to the simulation based training models. The benchmark has been established using knots tied by an experienced group of arthroscopic surgeons. On the simulation models they will continue training until proficiency has been demonstrated. Group C trainees will be supervised and given formative feedback on their performance by an experienced trainer and assessor during these skills training sessions. Group B will complete the same curriculum as Group C with the exception that they will be free to progress through training at their own desired pace with no requirement to demonstrate proficiency in any phase of the training.

Outcome assessment On completion of the course proper all three groups of trainees will be required to complete a Bankart procedure on a fresh cadaver specimen. They will be assisted by an experienced orthopaedic surgeon blinded to the subjects training group and instructed to assist ONLY at the direction of the study participant. No guidance will be provided. Their performance will be video recorded for subsequent analysis. Video raters will be experienced orthopaedic surgeons who will be blinded as to subjects' identity and group. Objective assessment of their intra-operative performance will be measured using the pre-defined performance metrics agreed upon by the Arthroscopy Association of North America Delphi panel and previously demonstrated to show construct validity. These will include correct steps of the procedure completed, time to complete the procedure, intra-procedure errors and sentinel errors.

Power Calculations Based on data derived from the construct validity study on the shoulder model the following power calculation estimates were obtained for Steps completed correctly, Time to perform the procedure, Errors and Sentinel Errors made. Calculations are based on a group size of n = 16, and reference mean and standard deviation scores, an Alpha of 5% which corresponds to a 95% Confidence Interval and a two-tailed statistical test for significance.

In previous studies a transfer effectiveness ration (TER) of 26 - 42% has been shown for novices.[13] A TER of 20% means that the PBP trained group would complete 44.88 steps of the procedure in comparison to the control groups 37.4. The statistical power of this difference is 0.99. A similar magnitude of difference (i.e., 20%) for time and sentinel errors would result in a mean difference with a statistical power > 0.8. Error scores would require a difference between the mean scores of the group ~ 40% to reach an acceptable level of significance. Furthermore, all of these differences would mean that the PBP trainees were still not outperforming the experts.

Statistical Analysis

The following analyses will be conducted:

1. The primary analysis will compare the intra-operative performance of trainees. For each subject, the number or intra-operative steps completed, time taken, errors and sentinel errors will be compared across the three groups with one factor analysis of variance (ANOVA) and specific contrasts between groups will be conducted with Scheffe F-tests.
2. Two-factor ANOVA for repeated measures where factor one is training method and factor two is the anchor number will be conducted on the variables listed. The sphericity assumption, relating to the conditional variances of all pair-wise differences will be evaluated.

Statistical comparisons for repeated measures for the placement of Anchors 1 - 3.

• Anchor Preparation and Insertion steps
• Anchor Preparation and Insertion Errors
• Suture Delivery and Management Steps
• Suture Delivery and Management Errors
• Knot Tying Steps
• Knot Tying Errors

Baseline Aptitude and Performance Pearson Product Moment Correlation Coefficient and Multiple Regression Analysis will be used to assess the strength of the relationship between trainees' performance on individual and combined baseline assessments and objectively assessed performance on the Bankart procedure. The same type of analysis will be used to assess the strength of the relationship between knowledge at the start of the course and Bankart procedure performance.