Testing the Accuracy of a Digital Test to Diagnose Covid-19

The Covid-19 viral pandemic has caused significant global losses and disruption to all aspects of society (including health, education, and business and economic security). One of the major difficulties in controlling the spread of this coronavirus has been the delayed and mild (or lack of) presentation of symptoms in infected individuals. Moreover, there is insufficient Covid-19 testing capacity in the UK, and only moderate accuracy of such tests at confirming coronavirus infection. Together, these obstacles have led to countless unknown coronavirus cases going unobserved and fuelling the viral spread in the population, by compromising the stringency of self- isolation measures undertaken by infected individuals who may have otherwise curbed or prevented their transmission of the virus. The profound and widespread cost of the continuing Covid-19 progression, coinciding with the lack of testing capacity, warrants the development of alternative diagnostic tools that reliably assess Covid-19 infection in the early stages of infection, while also being low- cost, low-burden, and easily administered to a wide proportion of the population.

The free symptom-monitoring app 'Covid Symptom Study' was launched in mid-March by health technology start-up Zoe Global Ltd, and is currently being used in the UK, US and Sweden, with more than 2.7 million users in the UK alone who use the app to self-report their Covid-19 symptoms. Upon registering to use the app, users are asked to report demographic and phenotypic data such as age, sex, BMI, ethnicity, contact with infected individuals (through a healthcare professional capacity), smoking behaviour, existing health conditions, among other information. From then on, users are asked to report, on a daily basis, their presentation of symptoms attributable to Covid-19 (or lack thereof) through the use of app-administered questionnaires, thus enabling real-time tracking of disease progression across the UK. The app also allows users to report their Covid-19 test results, thus enabling the development of prediction algorithms based solely on self-reported user data to predict the presence of infection in untested users.

On behalf of Zoe Global Ltd, the UK Department of Health and Social Care with support from the UK's Chief Scientific Advisor has committed to test up to 10,000 app-users per week for infection with SARS-CoV-2 across England and Northern Ireland, for the purpose of rapidly improving the accuracy of symptom-based predictions. Similar testing allowance may follow in Scotland and Wales.

Symptomatic app-users will be asked to get tested for SARS-CoV-2 infection, using the popular swab and qRT-PCR technique, and asked to report their test results in the app, while continuing to log their symptoms.

This validation study, conducted at King's College London, aims to validate the sensitivity and specificity of machine learning models as a diagnostic tool that predicts infection with SARS-CoV-2 based on app-reported symptoms and phenotypic data, against the 'gold-standard' swab PCR-test, by utilising the Covid Symptom Study app as a research platform.

It is hypothesised that by training the symptom-based models using swab test results and through multiple model iterations following continuous data input from reporting and tested app users, predictions of infection will be made with considerable accuracy, thus enabling the Covid Symptom Study app to be used as a diagnostic tool that alleviates the strain of testing capacity in the UK while being easily accessible and posing low user burden.

Study Design:

Due to the rapidly developing and uncertain duration and intensity of the Covid-19 pandemic, the present study design is prospective and one that enables regular iteration on prediction models and continuous accumulation of validation data. The study consists of a series of phases, each lasting 14 days. Before the start of each phase (day 0), a set of machine learning models will be frozen and submitted for validation on data collected during this and subsequent phases.

Machine learning algorithms improve with increasing data. Therefore, validation phases will continue as long as tests are available and app users consent to joining the study. Due to the uncertainty around the progression of UK infection rates, the validation study will be continue whilst it is of value to public health.

A detailed statistical analysis plan is described in the document attached to this record. A record of all machine learning models used for validation will be regularly updated on GitHub (https://github.com/zoe/covid-validation-study).