URAISE: Ultrasound Regional Anaesthesia Interpretation Skill Evaluation

This is a prospective, multi-centre study leading to the identification of a validated, reliable and defensible tool for assessing ultrasound-guided regional anaesthesia (UGRA) image interpretation.

This study will have 5 Phases:

Phase 1 - nerve block identification:

A panel of academic experts in regional anaesthesia will review published evidence from UGRA international guidelines. They will determine which UGRA nerve blocks are relevant to anaesthetic practice in the UK, and list the anatomical and clinical knowledge that is relevant to performing these nerve blocks.

Phase 2 - content and question generation:

A panel of experts in regional anaesthesia, recognised for their involvement in regional anaesthesia societies and publications, will review the UGRA nerve blocks and the relevant anatomical structures identified in Phase 1. The anatomical structures that must be identified on ultrasound for the safe and effective performance of these blocks will be agreed upon following panel discussion. Each structure will be categorised as either:

• Essential
• Desirable
• Expert
• Not relevant

This information will be collated by the Primary and Co-investigators to generate a pilot test of UGRA interpretation content and questions.

Phase 3 - preliminary testing:

The pilot test will be administered by local investigators via a computer-based application to 35-50 anaesthetic trainees and consultants, recruited by email and flyers sent to schools of anaesthesia across the UK. The cohorts will include an equal number of trainees and consultants of varying regional anaesthesia and clinical experience. The pilot test will contain short video clips and images of ultrasound guided regional anaesthesia with questions in a constructed response question (CRQ) format to identify relevant structures, anatomy and clinical knowledge, which will require the participant to write free-text responses to questions. It is important to recognise that some participants will be familiar with the CRQ format while others might be more familiar with other forms of assessment question. Participants in the pilot will therefore be provided with sample questions and answers prior to the test to familiarise themselves with the question format.

"Face validity" of the test (i.e. whether it measures what it claims to) will be assessed by asking participants to comment on ease of understanding and to identify terms that appear ambiguous. Participants will be able to comment in a text box after each item in the questionnaire. They will also be asked to comment on how they would improve the test and any relevant anatomy or clinical knowledge missing from the questions. The format of the test will be amended based on the comments.

A "Difficulty Index" for each item will be calculated by dividing the number of correct responses by the total number of responses. A higher value suggests a question is easier to answer. Any item with a difficulty index of < 50% will be considered too difficult to answer by the respondents and will be considered for removal. This will be compared to the information gathered from the panel of experts on test items (that is, whether the anatomical structure is essential, desirable, expert or not relevant to the specific ultrasound guided nerve block).

The pilot study will be used to identify how well the answers for each test item discriminate between the participant's level of ultrasound experience ("construct validity"). Differences in ultrasound interpretation skills between study groups will be analysed using statistical modelling. Items that discriminate well will be retained and those that discriminate poorly will be removed from the main test.

Reliability of the test will be improved by presenting the pilot test items in a random order to minimise the effect of cognitive fatigue diminishing performance on subsequent items. The average time taken for participants to complete each item in the pilot test will also be recorded to guide the optimal final study test length. It is important to balance the maximum number of test items whilst minimising the risk of cognitive fatigue and participant drop out. While the pilot test will necessarily have more items than the final test, pilot test participants will be asked to comment on the point at which they felt fatigued to guide this balance. It will also guide understanding of whether the level of regional anaesthesia experience affects fatiguability.

Phase 4 - testing for content validity:

To ensure all items in the assessment tool contain content that tests the domain the investigators intend to test, following the pilot the amended test will be presented to the panel of 5 experts in regional anaesthesia, who will be asked to rate the suitability of ultrasound images and questions to performance of the block. This will be on a 4-point Likert scale:

1. = definitely not suitable
2. = probably not suitable
3. = probably suitable
4. = definitely suitable The "content validity" (i.e. the extent to which a measure fairly represents a particular domain) will be derived by dividing the number of experts who rated the item as probably suitable and definitely suitable (3 and 4 on the scale) by the total number of experts. Items found to be appropriate by a sufficiently high percentage of participants will be selected.

Phase 5: psychometric main study testing The final assessment tool will be administered to an equal number of anaesthetists of varying regional anaesthesia experience. Participants will be recruited using emails and flyers sent to schools of anaesthesia across the UK. Each participant will answer a series of demographic questions, including age, gender, handedness, as well as level of training and practical experience in UGRA to account for inconsistencies in clinical exposure at different stages of training.

The exam will consist of constructed response questions (CRQs). Participants will be provided with sample questions and answers before the test to familiarise themselves with the question format. The construct validity of the tool will be tested by comparing the scores of anaesthetists with their level of experience in regional anaesthesia. The level of experience is a proxy outcome measure for their proficiency in ultrasound-guided regional anaesthesia. Differences in ultrasound interpretation skills between study groups will be analysed using statistical modelling. At the end of the assessment, there will be a short questionnaire to ascertain ease of understanding and suitability of ultrasound images to support the face validity of the test.

The "internal reliability" (i.e. the consistency of results across items in a test) of the responses will be assessed using statistical modelling. A split-test method will also be used for each participant to compare the first half of their assessment with the second half to ensure internal consistency of their item responses. The difficulty index calculated for items in the pilot study will be used to ensure a balanced set of test items, in terms of their difficulty, in the first and second halves of the test. All items will be made mandatory. The test will be administered by local investigators via a web-based application. It will have an automatic system of marking to minimise inter-marker differences. It will also be manually checked by investigators who are blinded to the level and identity of the participant.

The investigators will aim to recruit a minimum of 250 participants, each of whom will be allocated a participant number to maintain anonymity.

The results will be analysed only when recruitment has ended.