If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Orbis International, New York, New YorkCentre for Public Health, Queen’s University Belfast, Belfast, United KingdomZhongshan Ophthalmic Center, Sun Yat-sen University, Guangzhou, China
This trial was designed to determine if artificial intelligence (AI)-supported diabetic retinopathy (DR) screening improved referral uptake in Rwanda.
Design
The Rwanda Artificial Intelligence for Diabetic Retinopathy Screening (RAIDERS) study was an investigator-masked, parallel-group randomized controlled trial.
Participants
Patients ≥ 18 years of age with known diabetes who required referral for DR based on AI interpretation.
Methods
The RAIDERS study screened for DR using retinal imaging with AI interpretation implemented at 4 facilities from March 2021 through July 2021. Eligible participants were assigned randomly (1:1) to immediate feedback of AI grading (intervention) or communication of referral advice after human grading was completed 3 to 5 days after the initial screening (control).
Main Outcome Measures
Difference between study groups in the rate of presentation for referral services within 30 days of being informed of the need for a referral visit.
Results
Of the 823 clinic patients who met inclusion criteria, 275 participants (33.4%) showed positive findings for referable DR based on AI screening and were randomized for inclusion in the trial. Study participants (mean age, 50.7 years; 58.2% women) were randomized to the intervention (n = 136 [49.5%]) or control (n = 139 [50.5%]) groups. No significant intergroup differences were found at baseline, and main outcome data were available for analyses for 100% of participants. Referral adherence was statistically significantly higher in the intervention group (70/136 [51.5%]) versus the control group (55/139 [39.6%]; P = 0.048), a 30.1% increase. Older age (odds ratio [OR], 1.04; 95% confidence interval [CI], 1.02–1.05; P < 0.0001), male sex (OR, 2.07; 95% CI, 1.22–3.51; P = 0.007), rural residence (OR, 1.79; 95% CI, 1.07–3.01; P = 0.027), and intervention group (OR, 1.74; 95% CI, 1.05–2.88; P = 0.031) were statistically significantly associated with acceptance of referral in multivariate analyses.
Conclusions
Immediate feedback on referral status based on AI-supported screening was associated with statistically significantly higher referral adherence compared with delayed communications of results from human graders. These results provide evidence for an important benefit of AI screening in promoting adherence to prescribed treatment for diabetic eye care in sub-Saharan Africa.
The growing burden of diabetes and its associated complications is increasing the demands on health care systems, particularly in low-resource countries. Globally, the number of people living with diabetes is increasing rapidly, with the largest projected increase in Africa, an estimated 143% by 2045.
As reported by the Vision Loss Expert Group, the prevalence of DR is increasing in many regions globally, with the largest increase in southern sub-Saharan Africa.
GBD 2019 Blindness and Vision Impairment Collaborators, Vision Loss Expert Group of the Global Burden of Disease Study. Causes of blindness and vision impairment in 2020 and trends over 30 years, and prevalence of avoidable blindness in relation to VISION 2020: the Right to Sight: an analysis for the Global Burden of Disease Study.
Diabetic Retinopathy Study Research Group Photocoagulation treatment of proliferative diabetic retinopathy: clinical application of diabetic retinopathy study (DRS) findings, DRS report number 8.
Furthermore, where screening programs exist in sub-Saharan Africa, many patients identified with referable DR fail to comply with follow-up appointments.
Poor adherence to treatment and follow-up recommendations because of financial barriers, travel time, lack of clarity in the referral process, and uncertainty among patients about the treatability of the disease remain a significant barrier to positive clinical outcomes and preventing vision loss in patients with diabetes.
Recent advances in computer-based analysis using artificial intelligence (AI) present a promising opportunity to test and refine automatic grading of diabetic retinal images for screening. Using validated AI algorithms instead of scarce trained specialists could potentially increase the efficiency and accessibility of screening programs.
Systematic reviews of deep learning-based algorithms in DR screening have highlighted such advantages as reduction in demands for manpower, cost of screening, and intragrader and intergrader variability.
However, evidence from real-life screening programs is limited, and no evidence exists on community acceptance of the use of AI-supported DR screening.
Studies on the use of AI to screen for DR in Africa report good accuracy of the technology
Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study.
; however, the continent lags in deployment of AI in clinical settings. The present trial, piggybacking on an AI-based DR screening and service-delivery project in Kigali, Rwanda, was designed to assess whether use of Orbis International’s Cybersight AI in diabetes clinics leads to increased patient uptake of DR referral services. Our hypothesis was that adherence to referral services would be higher among patients randomized to receive AI-supported screening with immediate feedback compared with those randomized to receive delayed communication of results until after human grading was completed.
Methods
Study Design and Participants
The Rwanda Artificial Intelligence for Diabetic Retinopathy Screening (RAIDERS) study was an investigator-masked, parallel-group randomized controlled trial enrolling participants at 4 clinical sites in Rwanda. The study was approved by the Rwanda National Health Research Committee (identifier, NHRC/2020/PROT/025) and the Rwanda National Ethics Committee (identifier, 945/RNEC/2020). All participants provided written informed consent before enrollment. The tenets of the Declaration of Helsinki were followed throughout. The trial was registered on the Pan African Clinical Trial Registry (www.pactr.org; registry number, PACTR202101512465690).
Screening for DR using retinal imaging with AI interpretation was implemented from March 2021 through June 2021 at 4 diabetes clinics in and around Kigali, Rwanda (2 district-level clinics, 1 referral-level clinic, and 1 diabetes association-led clinic). Follow-up at the referral site continued through July 2021. Participants were recruited during routine visits to the diabetologist. The diabetologist or other attending clinician presented the patient information sheet and consent form to all potentially eligible patients before DR screening.
The study inclusion criteria were diagnosis of type 1 or 2 diabetes, ≥ 18 years of age, provision of informed consent, availability of gradable digital retinal images for ≥ 1 eye, willingness and ability to travel to the designated referral clinic, and lack of any current eye treatment or participation in any ongoing study requiring regular appointments for eye care. Additionally, all participants in both study groups were required to have a positive finding of referable DR or other condition requiring referral for additional investigations according to Cybersight AI. Patients with known DR, currently under the care of an eye doctor, participants who did not provide consent, and those with ungradable images received appropriate clinical care, but were excluded from the study.
Sample size estimates were based on obtaining data for the primary outcome: uptake of referrals within 30 days of receiving positive screening results. Using an uptake of eye examinations of 35% in the control group and 60% in the treatment group, a power of 90%, and a 2-sided α value of 5%, the estimated target sample size was 79 participants in each group. However, we continued to 275 total participants (137 in each group) to allow us to detect an uptake difference of 20%. With an expected screening positivity rate of 33%, we aimed to enroll 825 participants for the study.
AI Model
The model is based on the Inception ResNet version 2 convolutional neural network architecture
trained to classify fundus photographs into 1 of 5 categories based on the International Clinical Disease Severity Scale for DR (nonproliferative normal, nonproliferative mild, nonproliferative moderate, nonproliferative severe, and proliferative).
The input to the convolutional neural network is a single (or batch thereof) fundus photograph. Preprocessing includes: (1) removal of black fundus border, if present, and (2) resizing of the image to 448 × 448 pixels with preserved aspect ratio. The output of the model is an L1-normalized vector (i.e., sums to 1), where each element corresponds to 1 of the DR grades. The predicted DR grade is the argmax of the raw output vector, and the referable DR score is the sum of the last 3 elements of the output vectors.
The model was trained on a total of 90 073 photographs that were quality controlled by ≥ 1 board-certified ophthalmologist. Training was carried out “from scratch” based on randomly initialized weights. To improve generalization to unseen images, data augmentation techniques such as random zoom, flipping, and rotation were applied during training.
Referable DR performance validation carried out based on a balanced hold-out dataset (200 photographs per DR grade for a total of 1000 photographs) and based on an external benchmark dataset resulted in areas under the receiver operating receiver characteristic curve of 96% and 98.5%, respectively.
Participants’ baseline demographic data and clinical characteristics were collected on electronic devices using KoBoToolbox before retinal imaging. After imaging, a questionnaire was administered to all participants inquiring about satisfaction with the screening process and their eye care history and knowledge. Trained personnel captured 2-field (optic disc and center-cantered) digital color, nonstereo, nonmydriatic 45° retinal fundus photographs of each eye (Topcon NW400; Topcon). Retinal images were captured in the JPEG (joint photographic experts group) format, with a dimension of 2592 × 1944 pixels. If image quality was deemed poor because of a small pupillary aperture (< 2.5 mm), the eye was dilated with a single drop of tropicamide 0.5%, and the image was reacquired after 15 minutes.
After imaging was completed, all images, anonymized with a unique patient registration number, were uploaded to Orbis International’s Cybersight AI. A mobile device or laptop and an internet connection are required to access Cybersight AI, which generates a response regarding the presence or absence of referable DR based on a macula-centered image from each available eye of a participant within 60 seconds. The system automatically confirms that each image contains the correct features and is of sufficient quality for grading. Cybersight AI is available free of charge to eye health professionals in low- and middle-income countries and is accessible on completion of no-charge registration on Orbis International’s telehealth platform, Cybersight. All images also were uploaded to Labelbox (Labelbox, Inc) for grading by a United Kingdom National Health System formally trained retinal specialist.
Randomization and Masking
After imaging and interpretation of images by AI, eligible participants whose screening results were positive for referral by AI were randomly assigned (1:1) to either grading by AI with immediate feedback (intervention) or grading by human graders with communication of need for referral only after human grading was completed in 3 to 5 days (control). The study group was assigned for each participant by having them flip a coin that read “AI” on one side and “Human” on the other side. A few participants reluctant to toss a coin were randomized using sealed, opaque envelopes that had either “AI” or “Human” written on a sheet. The decision according to the AI system was used to determine referral for both study groups to guarantee that they would be similar at baseline, although only participants randomized to the intervention (AI) group received their reports immediately.
Clinic staff capturing and uploading retinal images, and those collecting outcome data at the referral site (such as receiving clerks entering attendance data), were masked to participant group assignment. All images from potential participants, regardless of the AI grade determining enrollment in the study, were graded by human experts masked to the AI grade, and thus participant inclusion or group assignment. For practical reasons, study participants, the study coordinator, and study personnel responsible for interviews and randomization were not masked; however, participants remained unaware of the study hypothesis and primary outcome.
Participants randomized to the intervention (AI) group were made aware that their screening report was automatically generated by the AI platform. No additional education on the AI system was provided. However, the intervention group was aware that their images would also be reviewed by human graders. Intervention participants received a report that included their fundus images and was color coded for severity of the DR grade (green, no DR; yellow, mild DR; orange, moderate DR; and red, vision-threatening DR). At this time, intervention participants were informed that referral to a secondary clinic for further ocular examination was required. Additional referral criteria included a cup-to-disc ratio of > 0.7 or any macular anomaly.
Participants in the control (human grading) group were unaware of the AI report, but were aware that they would be contacted about whether to follow-up at the eye clinic through short message service (SMS) and also through a phone call from the health worker who attended them, after the human grading report was completed in 3 to 5 days. Only after screeners had received human grading reports were control participants informed that they needed to visit the referral clinic. When AI findings had been positive (patient thus recruited into trial) and human grading results were negative, the patient remained in the trial and was informed of the status after the full follow-up examination.
Participants in both study groups received clear instructions about the follow-up process, including the location of the eye clinic and information on reimbursement of travel costs and insurance copayments. The referral site was a secondary-level clinic with an ophthalmologist skilled in the management of DR. At the time of receiving the report on the examination (immediately for the intervention group, or after 3 to 5 days for the control group), participants were informed that they could report to the eye clinic on any working day within the next 30 days.
Assessment of Outcomes
The main study outcome is the difference between study groups in the rate of presentation for recommended referral services within 30 days of being informed of the need for a referral visit. Attendance for eye examination at the eye clinic was recorded on an outcome form by an independent research assistant who was masked to participant study group assignment. The project manager received the completed outcome forms and linked the data to the participant database for each participant. Nonattendance was defined as failure to attend the referral clinic on any occasion within 30 days of being recommended for follow-up. Participants in the control group who could not be successfully contacted by SMS or telephone (n = 5) were still included in the denominator as requiring referral.
Statistical Analysis
Patients with nondeferrable outcomes on images according to the AI were not enrolled in the trial and were excluded from the analyses. Demographic and clinical characteristics are presented stratified by study group according to the principle of intention to treat. Frequencies and percentages for categorical variables and means and standard deviations for continuous variables are presented, stratified by study group. Responses of “I don’t know” regarding awareness of eye care knowledge and beliefs are categorized as negative or incorrect responses. Rates of attendance at the referral clinic visit (main outcome) are described as unadjusted percentages, comparing the 2 study groups using the chi-square test. Univariate and multivariate logistic regression modeling was used to compare the main outcome between study groups and other potential predictors. Variables that were significant in the univariate models were included in the multivariate model if they remained significant (P < 0.05).
Role of the Funding Source
The project was supported by Orbis International and the Association for Research in Vision and Ophthalmology (Roche Award). The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.
Results
Of the 827 clinic patients with diabetes who gave consent and were screened for DR, 4 did not meet inclusion criteria, specifically the ability to travel. A total of 275 participants (33.4%) met the referral criteria on AI screening and were randomized for inclusion in the trial, with 136 (49.5%) allocated to the intervention group and 139 (50.5%) allocated to the control group. All participants (100%) in each study group received the allocated intervention, and data for the main outcome was available for all (100%) participants (Fig 1).
Figure 1Flowchart showing Rwanda Artificial Intelligence for Diabetic Retinopathy Screening trial enrollment. AI = artificial intelligence; DR = diabetic retinopathy.
Baseline demographic data (Table 1) did not differ significantly between study groups. Satisfaction with the screening process was very high in both groups (intervention, 100%; control, 99.3%). A total of 5 participants (3.60%) in the control group could not be contacted by SMS or telephone, but were retained in the analysis as specified pre hoc in the registered protocol. None returned for follow-up.
Table 1Baseline Demographic and Clinical Characteristics by Study Group
Status based on official Ubudehe classification that exists for all Rwandans (https://rwandapedia.rw/hgs/ubudehe/poverty-level-categories) and that was reviewed in 2020. Category A is highest, categories B and C are combined into medium, and categories D and E are the lowest. No statistically significant differences exist between the two study groups.
Highest
18 (13.2)
14 (10.1)
Medium
99 (72.8)
113 (81.3)
Lowest
4 (2.9)
4 (2.9)
Unknown
15 (11.0)
8 (5.8)
Health insurance
None
6 (4.4)
6 (4.3)
Public
117 (86.0)
124 (89.2)
Private
8 (5.9)
4 (2.9)
Other
5 (3.7)
5 (3.6)
Occupation
Professional
8 (5.9)
10 (7.2)
Skilled work
31 (22.8)
27 (19.4)
Unskilled work
17 (12.5)
18 (13.0)
Unemployed
71 (52.2)
80 (57.6)
Retired/pensioner
9 (6.6)
4 (2.9)
Diabetes status
Type of diabetes
1
63 (46.3)
56 (40.3)
2
71 (52.2)
80 (57.6)
Unknown
2 (1.5)
3 (2.2)
Duration (yrs)
<5
47 (34.6)
36 (25.9)
5–10
43 (31.6)
45 (32.4)
>10
46 (33.8)
58 (41.7)
Blood glucose (mg/dl)
8.97 ± 3.69 (n = 123)
10.3 ± 5.01 (n = 120)
Eye care history and knowledge
Patient reports dilated eye examination in past year
14 (10.3)
23 (16.6)
Aware diabetes can cause eye problems
121 (89.0)
125 (89.9)
Personally knows a blind person
64 (47.1)
64 (46.4)
Worried about losing sight
110 (80.9)
112 (80.6)
High satisfaction with screening processes
136 (100)
137 (99.3)
Mean distance of home from referral site (km)
17.2 ± 20.1
15.8 ± 18.4
Residence
Urban
79 (58.1)
85 (61.2)
Rural
57 (41.9)
54 (38.8)
Data are presented as no. (%) or mean±standard deviation.
∗ Status based on official Ubudehe classification that exists for all Rwandans (https://rwandapedia.rw/hgs/ubudehe/poverty-level-categories) and that was reviewed in 2020. Category A is highest, categories B and C are combined into medium, and categories D and E are the lowest. No statistically significant differences exist between the two study groups.
Adherence with recommended referral examination was significantly higher in the intervention group (70/136 [51.5%]) versus the control group (55/139 [39.6%]; P = 0.048) in unadjusted analyses (Table 2), representing a 30.1% increase. When adjusted for age, sex, and urban or rural residence, membership in the intervention was significantly associated with acceptance of recommended referral (odds ratio, 1.74; 95% confidence interval, 1.05–2.88; P = 0.031; Table 3). Participants in the AI group also sought treatment at the referral clinic much sooner after receiving referral advice than those in the control group: AI group, 6.6 ± 7.4 days (median, 4.0 days; range, 0–30 days) versus control group, 9.6 ± 5.1 days (median, 8.0 days; range, 3–28 days; P < 0.0001, Wilcoxon rank-sum test).
Table 2Unadjusted Comparison of Primary Outcome between the Intervention and Control Groups
but they are dependent on adherence with referral services to succeed in preventing negative clinical outcomes. Previous studies have reported successful DR screening programs based in primary care and diabetes clinics,
Real-world artificial intelligence-based opportunistic screening for diabetic retinopathy in endocrinology and indigenous healthcare settings in Australia.
Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study.
the current study focused on adherence with screening referral as a crucial step in the practical application of AI-supported DR screening in low-resource settings, yet the RAIDERS study strengthens the evidence because of its randomized clinical trial design. The RAIDERS trial found increased adherence to DR referral services among participants randomized to receive immediate feedback on referral status based on AI-supported screening compared with those randomized to receiving referral advice only after the human grading report was completed.
Data on adherence with follow-up after DR screening in low-resource countries are limited. The adherence rate for the intervention (AI) group in this study (51.5%) is higher than rates reported in traditional screening programs not delivering immediate feedback on need for referral, including a study in the neighboring country of Tanzania (25%),
have been reported. Artificial intelligence-supported screening provides an opportunity for immediate counselling and eye health education for those requiring referral, potentially contributing to increased adherence. Other studies in low-resource settings have reported improved adherence with referral care in response to eye health education.
Patients’ adherence to recommended follow-up eye care after diabetic retinopathy screening in a publicly funded county clinic and factors associated with follow-up eye care use.
Provision of an instant report that includes images of the retina may support acceptance of the recommended referral, especially for those who are asymptomatic. Furthermore, in this model, participants who wished to visit the ophthalmologist on the same day as screening could do so, potentially minimizing travel and accounting for the quicker uptake of referral seen in this study, although additional research on factors contributing to increased adherence is needed. The study by Watane et al
found that longer recommended intervals for follow-up led to increased odds of follow-up nonadherence. We hypothesized that an important reason for lower acceptance of care in the control group would be an inability to contact these participants after they left the clinic, but in fact, nonadherence for this reason was not common. Follow-up rates in the control group (39.6%) were higher than those in the intervention group in Tanzania, potentially because of high insurance coverage rates in Rwanda, which reduced barriers of cost, and the impact of phone reminders to relay the results of the DR screening. Such contact with SMS reminders
has been shown to increase adherence with recommended eye care.
Although an increase in the uptake of referral services was observed in the intervention group compared with the control group in the current study, close to half of those undergoing AI screening did not comply with referral care within 30 days. Clearly, other interventions in combination with AI-supported screening are needed to increase adherence further.
Strengths of the study include results that are broadly applicable across AI-supported DR screening models, regardless of the AI platform used. The trial was a randomized design with good fidelity to protocol and high follow-up rates. Additionally, drawing participants from different types of facilities provided data representative of the broader Rwandan health system. Furthermore, the study provides evidence on the practical application of AI-supported DR screening in low-resource settings, specifically in Africa, where few trial data on the programmatic impact of AI exist. Key factors that should be considered when thinking about a similar approach in other locations include support from the diabetes care providers to provide time and space for eye screening, the availability and accessibility of an accurate AI system, using a high-quality nonmydriatic camera for imaging acquisition, and ensuring confidence in the system by those presenting it to patients. Acknowledging limitations, the study included a relatively higher proportion of patients from urban areas (60%), meaning that the application of the results to rural settings must be made with caution. Accuracy of the AI platform used has not been validated in a peer-reviewed publication for those of African descent, but the accuracy of the referral was not relevant to the main trial outcome, and overread precautions were implemented. Finally, it was impractical to mask participants, but placebo effects are not of particular concern for the main outcome of this study.
In conclusion, this study demonstrated the potential of AI-supported DR screening to deliver increased uptake of referral services. Results of the RAIDERS trial provide evidence for the integration of AI for DR screening as part of a sustainable national eye care program to prevent DR-related blindness in sub-Saharan Africa. Additional research on methods to further increase adherence with referral services is needed, because early diagnosis and treatment of DR is highly effective in preventing vision loss.
Diabetic Retinopathy Study Research Group Photocoagulation treatment of proliferative diabetic retinopathy: clinical application of diabetic retinopathy study (DRS) findings, DRS report number 8.
Further research on the use of AI-supported DR screening in low- to middle-income countries is also necessary to better understand how to integrate this technology into efficient DR care systems in low-resource settings.
References
International Diabetes Federation. International Diabetes Federation diabetes atlas, 9th ed. Available at: https://www.diabetesatlas.org; 2019 Accessed 20.08.21.
GBD 2019 Blindness and Vision Impairment Collaborators, Vision Loss Expert Group of the Global Burden of Disease Study. Causes of blindness and vision impairment in 2020 and trends over 30 years, and prevalence of avoidable blindness in relation to VISION 2020: the Right to Sight: an analysis for the Global Burden of Disease Study.
Photocoagulation treatment of proliferative diabetic retinopathy: clinical application of diabetic retinopathy study (DRS) findings, DRS report number 8.
Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study.
Real-world artificial intelligence-based opportunistic screening for diabetic retinopathy in endocrinology and indigenous healthcare settings in Australia.
Patients’ adherence to recommended follow-up eye care after diabetic retinopathy screening in a publicly funded county clinic and factors associated with follow-up eye care use.
All authors have completed and submitted the ICMJE disclosures form.
The author(s) have made the following disclosure(s): W.M.: Employee – Orbis International
N.W.: Employee – Orbis International
G.L.: Employee – Orbis International
M.Y.K.: Royalties – New World Medical, SpyGlass Pharma
D.H.C.: Employee – Orbis International
N.C.: Employee – Orbis International; Consultant – Belkin Vision
N.J.: Employee – Orbis International
Supported by Orbis International, New York, New York; and Association for Research in Vision and Ophthalmology (Roche Award).
HUMAN SUBJECTS: Human subjects were included in this study. The human ethics committees at the Rwanda National Health Research Committee and the Rwanda National Ethics Committee approved the study. All research adhered to the tenets of the Declaration of Helsinki. All participants provided informed consent.
No animal subjects were included in this study.
Author Contributions:
Conception and design: Mathenge, Nkurikiye, Uwaliraye, Kahook, Cherwek, Congdon, Jaccard
Analysis and interpretation: Mathenge, Whitestone, Patnaik, Piyasena, Lanouette, Congdon, Jaccard
Data collection: Mathenge, Nkurikiye, Uwaliraye, Lanouette, Cherwek, Congdon, Jaccard
Obtained funding: N/A; Study was performed as part of regular employment duties at Orbis International. No additional funding was provided.
00057-4&id=gr1.jpg){kind=link}