Diabetes Screening and Monitoring Using Tongue Images and Self-reported Symptoms: a Machine Learning Approach

Tongue image collection Tongue image collection will be carried out by trained research assistants based on our published protocol (Wang et al., submitted), which consisted of the following steps. Step 1, start the phone's camera function without using any filter or artificial intelligence (A.I.) function; Step 2, turn on the flash; Step 3, aim at the lips of the subject by holding the phone directly in front of, and 15-20 cm away from, the subject with a 45 degree angle; Step 4, tap on the screen to adjust focus when the subject protrudes the tongue and then take a picture; Step 5, upload the image in full resolution to the online questionnaire system (Qualtrics survey software).

Chinese medicine questionnaire collection A questionnaire in electronic/paper form will be used to collect symptoms relevant to Chinese medicine pattern diagnosis of diabetes. The questions have been selected from previous studies on the most commonly appearing symptoms of diabetes from Chinese medicine perspectives (Hsu et al., 2016; Zhou et al., 2012; Zhao et al., 2017). Age, gender, weight, height , duration of diabetes, family history of diabetes and any comorbid disease, Hb reading (if available) will also be taken into consideration. Recent history of smoking is also recorded as a potential confounding factor for tongue coating features (Tomooka et al., 2017)

Data entry and verification To ensure accuracy, image, questionnaire answers and HbA1c (and Hb, if available) readings will be uploaded to the server at the same time. A patient identification number for the trial will also be generated to facilitate data verification. Such identification number consists of information on data collection date and outpatient number, etc. However, it will not contain any information related to patient's HKID, full patient name, HN, MRN, DOB, address or phone number.

The data will be collected by an online questionnaire service using Qualtrics and then store at a local computer at SCM/HKBU. A paper version of the questionnaire may be used when necessary, and the filled questionnaire will be kept for 3 years after the study. The collected image will contain no eye features on the photo, and the questionnaire will not include any information for identification of the subject, except a subject ID number for the trial. After completion of the project, data on Qualtrics will be deleted, whereas data on the local computer will be kept for seven years after the study has been published, as per institution requirement.

Machine learning Two approaches will be used in machine learning. In the first approach, we will first perform image classification of either normal or abnormal HbA1c and generate the probabilities for the classification using convolutional neural networks (CNNs) (Anwar et al., 2016; Ødegaard et al., 2016). Then we will try to classify the images into four different classes according to their HbA1c level: <6% (normal), 6-6.4% (prediabetes), 6.5-8.9% (diabetes) and ≥ 9% (diabetes with high HbA1c). The probability data of image classification will be combined with data from the questionnaire as variables using fully connected CNNs layers to determine which one of the four HbA1c levels the subject belongs to (Osia et al., 2018). The advantage of this approach is that it allow us to determine the sensitivity and specificity of the classifications using tongue image alone. However, a drawback of this approach is that it may lose some important features of the tongue in classification with questionnaire data. Alternatively, we will extract features from each tongue image using CNNs, and use one hot encoding to transfer the features into one dimensional matrix, which enable the blending of image data with questionnaire data for regression analysis using neural network.

A separate testing data set, which contains at least 200 negative and 200 positive images, will be used to evaluated sensitivity and specificity of the algorithms obtained. In the final algorithm, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) will be reported.

Potential problems and experimental alternatives

Potential problem 1:

It is highly likely that the tongue image features are not exclusive to abnormal level of HbA1c, and they are also present in smokers and some acute and chronic diseases, such as COPD, upper respiratory infection and chronic kidney or liver diseases.

Experimental alternatives:

We have now included a section for smoking history and known acute or chronic diseases in our questionnaire. If we found a significant correlation between certain confounding variables with abnormal HbA1c tongue diagnosis, we would warn the future user about the unreliability of the algorithm when used in a population with the confounding condition. Furthermore, we would adjust the weighting of parameters in the questionnaire to achieve a prediction model with reasonable specificity and sensitivity, even for populations with the said condition.

Potential problem 2:

Machine classification results from the collected tongue image data set show poor specificity and sensitivity in predicting the four HbA1c levels, even after the inclusion of parameters in TCM questionnaire.

Experimental alternatives:

As we will collect blood lipid profile of subjects from the hospital when available, we can analyze our results as we go. When our total sample population reaches 2,000, and we still see a poor specificity and sensitivity, we will include blood lipid profile test to our community subjects. By adding blood lipid profile as parameters in our regression model, we should be able to enhance the specificity and sensitivity of the prediction, as previous studies have found a close correlation between high level of HbA1c and dyslipidemia (Khan et al., 2007). Of course, we do not wish this to happen because it will lower the value of our prediction model, as it requires an invasive procedure to obtain blood lipid profile. Nevertheless, we are confident that at least a non-invasive model for detection of abnormal level of HbA1c (≥ 6.5%) can be produced, based on the results from our proof-of-concept study.

Potential problem 3:

The image features responsible for the classification of normal and abnormal levels of HbA1c (≥6.5%) could not be readily identified.

Experimental alternatives:

Extraction of visual features that can provide a semantic and robust representation of tongue images with abnormal levels of HbA1c is highly desirable. The greasy and thick coating, which we have identified as features for the classification, needs to be further defined. To this end, we will try different techniques proposed in the literature (Ching et al., 2018), including assigning example-specific importance scores, matching or exaggerating the hidden representation, activation maximization and latent space manipulation, etc.