Dataset collection and preparation for agent training
The pipeline for our study is shown in Fig. 1. The training dataset included clinical records of 594 CC patients before and after surgery, and 4881 of their follow-up images from January 2011 to December 2016. The eligible criteria of patients included diagnosis of CC, informed consent, and completed records of baseline information, lesion condition, comorbidities, surgical procedures, and complications in the first follow-up year. The eligible images for training were defined as the anterior segment images covering the posterior lens capsule by retro-illumination using slit-lamp photography. All these records were derived from routine examinations at the Childhood Cataract Program of the Chinese Ministry of Health (CCPMOH)18.
Functional architecture of CC-Guardian
The CC-Guardian incorporates three functional modules: a prediction module designed to identify high-risk patients who are likely to suffer complications (Fig. 2a), a dispatching module responsible for scheduling individual follow-up visits (depending on the results of the prediction module) (Fig. 2b), and a telehealth module that makes a clinical decision regarding additional treatment (intervention or continued follow-up) in each follow-up examination using telehealth computing (Fig. 2c).
Naive Bayes algorithm for prediction module
We selected 12 variables that both can be easily collected in routine clinical practice and were reported to be associated with complications after surgery16,17 as the inputs for the prediction module. These 12 variables can be generally grouped into four categories: (1) baseline information, (2) lesion condition, (3) comorbidities, and (4) surgical procedures (“Methods”).
To find the optimal algorithm, we first performed a comparison among naive Bayes and two classic machine learning algorithms (random forest and neural network, “Methods”). Based on the average performance of the 5-fold cross-validation by the training set of 594 CC patients, the naive Bayes algorithm provided comparative performance with random forest and neural network algorithms (Table 1). We finally selected the naive Bayes algorithm due to its intrinsic simplicity19, which can improve the processing speed and saving computing resource in real-world application. After selecting the naive Bayes, a complete algorithm was then trained on the entire dataset of 594 CC patients prior to later validations.
The normalized contributions of all 12 input variables to predict VAO (from 0.01 to 0.2, Supplementary Fig. 1a) and high IOP (from 0.03 to 0.15, Supplementary Fig. 1b) were presented, suggesting their substantial importance to outcome. Both the age at surgery (0.07 to VAO and 0.12 to high IOP) and gender (0.03 to VAO and 0.02 to high IOP) do not show any dominant role, indicating that predictive results will not significantly vary from these two variables.
Follow-up schedules for dispatching module
For patients predicted with no complications, the dispatching module will schedule seven time points (1 week, 1 month, 2 months, 3 months, 6 months, 9 months, and 12 months after surgery), which is the conventional plan in current clinical practice14. If the patients were predicted to have VAO, the module will add two time points (7.5 months and 10.5 months) to the conventional plan, since VAO was inclined to happen in the second half year17,20. If the patients were predicted to have high IOP, the module will add three time points (2.5 weeks, 2.5 months and 4.5 months) to the conventional plan, since high IOP was inclined to happen in the second half year16,20.
Deep residual network for telehealth module
A total of 4881 postoperative retro-illumination images were included as the training set for telehealth module. These 4881 images were collected from 2175 CC patients. Images from the same child were taken at different follow-up visits. Each image was independently labeled (intervention or continued follow-up) by two licensed ophthalmologists, and a third ophthalmologist was consulted when disagreements occurred. Then, a senior ophthalmologist with over 20 years of clinical CC experience verified the labels for each image. The ophthalmologists were blind and had no access to the outcome classifications. A stacked 101-layer residual network21 was utilized for telehealth module training and classification (“Methods”). The first 100 layers were used to extract multidimensional, high-level features from the input images and a Softmax classifier was applied to the last layer. The architecture of the deep residual network is provided in Supplementary Fig. 2 in the form of a diagram that highlights the arrangement of each layer.
A randomly selected dataset derived from the original CCPMOH records prior to training was used to validate CC-Guardian, which included clinical records of 142 patients (61 VAO, 81 non-VAO; 79 high-IOP, 63 normal) and 1220 follow-up images (671 follow-up, 549 intervention). The 1,220 images were collected from 507 CC patients. This dataset did not overlap with the CCPMOH data for agent training. The trained deep-learning model was frozen prior to any validations. The deep-learning predictions with time stamps were verified and saved by an individual who was blind to the expert-panel labels, thus precluding information leaking or double-dipping when comparing the prediction results with the expert-panel labels. After internal validation, the models trained on all of the training sets were used for following validation.
Using the prediction module, our agent predicted VAO with sensitivity of 0.967, specificity of 0.975, and area under curve (AUC) of 0.991; and high IOP with sensitivity of 0.962, specificity of 0.952, and AUC of 0.979. By virtue of its telehealth module, CC-Guardian provides intervention decisions with sensitivity of 0.991, specificity of 0.994, and AUC of 0.996. The detailed distributions for the accurate and mistaken detections are presented in Fig. 3a. Confusion matrices and receiver operating characteristic (ROC) curves are shown in Fig. 3b, c, respectively.
The multi-resource dataset had two parts. The first part consisted of clinical records of 79 patients (33 VAO, 46 non-VAO; 28 high-IOP, 51 normal) collected from 7 centers, which were used for validating prediction module. Seven centers included Shenzhen Eye Hospital, Beijing Tongren Hospital, People’s Hospital of Guangxi Zhuang Autonomous Region, the Third Affiliated Hospital of Sun Yat-sen University, Puning People’s Hospital, The Central Hospital of Wuhan, and the First Affiliated Hospital of University of South China. These 7 centers were distributed separately in northern, central, and southern China, and thus collectively composed a representative and independent sample for Chinese population. The eligible criteria were the same as the previous datasets, which were diagnosis of CC, informed consent, and completed records of inputs and outcomes required for prediction module. Detailed description of the datasets collected in each center was summarized in Supplementary Table 1.
The second part contained images collected from heterogeneous open-access databases, which can further validate versatility and generalization of telehealth module. We performed image searches using the Google, Baidu and Bing search engines through June 2016, with a combination of key words (e.g., congenital, infantile, pediatric cataract, follow-up, and retro-illumination) in the form of title words or medical subject headings. Two individuals (E.L. and H.L.) independently completed the searches. In addition, these two individuals cross-checked and confirmed that all the collected cases were retro-illumination images of CC during follow-up management. When discrepancies arose, consensus was achieved after further discussion. All confirmed images (91 follow-up and 123 intervention) were subsequently sent to CC-Guardian for validation.
The CC-Guardian’s prediction module predicted VAO with sensitivity of 0.940, specificity of 0.935, and area under curve (AUC) of 0.944; and high IOP with sensitivity of 0.964, specificity of 0.941, and AUC of 0.961. The CC-Guardian’s telehealth module provided intervention suggestions with sensitivity of 0.959, specificity of 0.945, and AUC of 0.981, a performance comparable to that obtained from internal validation. The detailed distributions for the accurate and mistaken detections are presented in Fig. 4a. Confusion matrices and ROC curves are shown in Fig. 4b, c, respectively.
CC-Guardian web-based smartphone app
To establish a telehealth cloud platform for clinical application, we built a CC-Guardian web-based smartphone app. The functions of the website (available at https://www.cc-cruiser.com/cc_guardian) include prediction-metric input, risk-stratification output, follow-up-examination results upload, and intervention decisions reporting. The functions of the smartphone app (available for iOS and Android systems) include a doctor portal (individual follow-up scheduling and updating) and a patient portal (automatic short message service (SMS) and interaction between patients and CCPMOH).
Users login to the website and input the necessary prediction metrics to obtain the risk-stratification output (occurrence of VAO and/or high IOP) (Supplementary Fig. 3a). Doctors can assign an individual follow-up plan using the smartphone app (doctor portal) (Supplementary Fig. 3b). Patients will receive individual follow-up schedules and can check these schedules using the smartphone app (patient portal) (Supplementary Fig. 3c). Interaction between patients and CCPMOH is enabled through either the website or the app. Doctors can also reschedule individual plans if necessary. During the follow-up process, users can upload their follow-up examination results to the website and obtain an intervention decision in a timely manner (Supplementary Fig. 3d). The algorithm embedded in this app is the same as the one in following validations.
CC-Guardian prediction-telehealth cloud platform
A compatible mechanism to promote the real-world clinical application of CC-Guardian to the follow-up management is highly desirable. Therefore, we created a prediction-telehealth cloud platform for the CC-Guardian smartphone app. As shown in Fig. 5, when a potential patient registers at the specialized care center (CCPMOH), their clinical metrics (valuable for the prediction module) are collected with their permission and immediately sent to the CC-Guardian website for complication prediction. Based on the prediction results, the dispatching module individually determines a follow-up schedule and sends an SMS to remind the patient in a timely manner. Patients can complete their regular follow-up examinations in primary care hospitals (telehealth follow-up examination) and upload their examination results to the web-based telehealth module. If the telehealth module recommends intervention, a fast-track notification system is triggered, and an emergency notification is sent to doctors at CCPMOH for immediate confirmation. Such patients will be informed that they should undergo intervention to manage their complications. Instructions for using the prediction-telehealth cloud platform are provided in Supplementary Video 1.
Retrospective self-controlled test
To further investigate the real-world efficiency of our system, we retrospectively used longitudinal follow-up records of 141 CC patients from CCPMOH between January of 2016 and May of 2017 (“Methods”). Before using our system, these 141 CC patients had 987 distant follow-up visits to CCPMOH, including 93 patients who underwent VAO and 105 patients who suffered from high IOP (Fig. 6a, b).
After applying our system, 90 cases of VAO (90/93, 96.8%) and 101 cases of high IOP (101/105, 96.2%) were successfully predicted (Fig. 6a, b). A total of 73 patients (73/90, 81.1%) would have benefited from earlier VAO detection (2759 risk days in total, 37.8 risk days earlier per person on average, median 45.0 risk days) and 89 patients (89/101, 88.1%) would have benefited from earlier high-IOP detection (1709 risk days in total, 19.2 risk days earlier per person on average, median 15.0 risk days).
All 141 patients would have had a total of 1579 telehealth visits rather than 987 distant visits to the CCPMOH, which would have, on average, saved each family significant travel of 928.6 miles per year (1185.4 vs. 256.8 miles, P < 0.001, Fig. 6c), time of 24.9 h per year (33.8 vs. 8.9 h, P < 0.001, Fig. 6d), and an expenditure of $1324.1 per year, ($1791.4 vs. $467.3, P < 0.001, Fig. 6e). These results demonstrate that our system has the potentials to detect and address the complications at an earlier stage, and to reduce the socioeconomic burden faced by patients compared to conventional in-person follow-up.