Improving the Accuracy of Artificial Intelligence Triage in Primary Care

Background GP practice staff triage patients contacting them to make the best use of resources and maintain patient safety. Online consultation systems are used by most GP practices and allow patients to contact their GP practice using an online form. They can be submitted without talking to a member of staff, thereby circumventing the usual triage process. Online consultation systems can triage patients using 'Artificial Intelligence' (AI), though there is a lack of research on their performance. We (The University of Manchester; UoM) propose to fill this gap by collaborating with an online consultation system provider with optional AI triage functionality (Patchs).

Research questions Overall research question: is it possible to develop AI models that can replicate clinicians' triage decisions?

1. What challenges do patients and GP practices face when triaging patients in primary care, and what are their drivers?
2. What is the best performing AI model for triaging patients in primary care?
3. Is AI triage performance maintained across different geographical regions?
4. Is AI triage performance maintained over time?
5. How does AI triage performance compare to current clinical practice?
6. Does AI triage performance change when deployed into clinical practice?
7. Does AI triage work fairly for all patients? Methods Workstream 1: Triage problem quantification. We will analyse anonymised historic data from GP practices using Patchs with AI triage disabled. Where publicly available, we will compare this to practice-level data from GP practices not using Patchs (control practices). We will undertake descriptive and inferential analyses to understand potential triage problems and factors that influence them, such as delays in providing patient care.

Workstream 2: AI development. We will use anonymised historic data from GP practices using Patchs to build new versions of the AI triage models currently in use with four different approaches: logistic regression, XGBoost, long short-term memory (LSTM), and large language model (LLM). We will use internal-external cross-validation by geographical region and compare their performance using random-effects meta-analysis and sub-group analyses to assess fairness (e.g. across ethnicities). We will compare their performance to the current AI triage models in use. The final version of the best-performing AI models will be developed using the entire dataset.

Workstream 3: Prospective background evaluation. We will obtain predictions from the best-performing AI models on prospectively collected data from GP practices using Patchs without AI triage by running the models in the 'background'. We will undertake sub-group analyses to assess fairness as described above.

Workstream 4: Prospective implementation evaluation. In accordance with the normal Patchs software updates, we will update the AI models in GP practices already using AI triage with the best-performing versions. We will prospectively measure how often GP practice staff and patients agree with the new versions' triage predictions to test whether its performance translates to real patient care. We will undertake sub-group analyses to assess fairness as described above.

Anticipated benefits We will help understand the problems currently faced by GP practices during online consultation triage. If we developed improved AI models, there may be improved patient safety (e.g. by helping patients receive help sooner) and reduced GP practice workload (e.g. by automating the triage process). GP practices and their patients in Workstream 4 would benefit immediately. We will provide evidence for GP practices not currently using AI triage whether to adopt it.