Advertisement

Impact of Artificial Intelligence Assessment of Diabetic Retinopathy on Referral Service Uptake in a Low-Resource Setting

The RAIDERS Randomized Trial
Open AccessPublished:April 29, 2022DOI:https://doi.org/10.1016/j.xops.2022.100168

      Purpose

      This trial was designed to determine if artificial intelligence (AI)-supported diabetic retinopathy (DR) screening improved referral uptake in Rwanda.

      Design

      The Rwanda Artificial Intelligence for Diabetic Retinopathy Screening (RAIDERS) study was an investigator-masked, parallel-group randomized controlled trial.

      Participants

      Patients ≥ 18 years of age with known diabetes who required referral for DR based on AI interpretation.

      Methods

      The RAIDERS study screened for DR using retinal imaging with AI interpretation implemented at 4 facilities from March 2021 through July 2021. Eligible participants were assigned randomly (1:1) to immediate feedback of AI grading (intervention) or communication of referral advice after human grading was completed 3 to 5 days after the initial screening (control).

      Main Outcome Measures

      Difference between study groups in the rate of presentation for referral services within 30 days of being informed of the need for a referral visit.

      Results

      Of the 823 clinic patients who met inclusion criteria, 275 participants (33.4%) showed positive findings for referable DR based on AI screening and were randomized for inclusion in the trial. Study participants (mean age, 50.7 years; 58.2% women) were randomized to the intervention (n = 136 [49.5%]) or control (n = 139 [50.5%]) groups. No significant intergroup differences were found at baseline, and main outcome data were available for analyses for 100% of participants. Referral adherence was statistically significantly higher in the intervention group (70/136 [51.5%]) versus the control group (55/139 [39.6%]; P = 0.048), a 30.1% increase. Older age (odds ratio [OR], 1.04; 95% confidence interval [CI], 1.02–1.05; P < 0.0001), male sex (OR, 2.07; 95% CI, 1.22–3.51; P = 0.007), rural residence (OR, 1.79; 95% CI, 1.07–3.01; P = 0.027), and intervention group (OR, 1.74; 95% CI, 1.05–2.88; P = 0.031) were statistically significantly associated with acceptance of referral in multivariate analyses.

      Conclusions

      Immediate feedback on referral status based on AI-supported screening was associated with statistically significantly higher referral adherence compared with delayed communications of results from human graders. These results provide evidence for an important benefit of AI screening in promoting adherence to prescribed treatment for diabetic eye care in sub-Saharan Africa.

      Keywords

      Abbreviations and Acronyms:

      AI (artificial intelligence), CI (confidence interval), DR (diabetic retinopathy), OR (odds ratio), RAIDERS (Rwanda Artificial Intelligence for Diabetic Retinopathy Screening), SMS (short message service)
      The growing burden of diabetes and its associated complications is increasing the demands on health care systems, particularly in low-resource countries. Globally, the number of people living with diabetes is increasing rapidly, with the largest projected increase in Africa, an estimated 143% by 2045.

      International Diabetes Federation. International Diabetes Federation diabetes atlas, 9th ed. Available at: https://www.diabetesatlas.org; 2019 Accessed 20.08.21.

      Diabetic retinopathy (DR), a complication of diabetes, is the leading cause of vision loss in working-age adults globally.

      International Diabetes Federation. Diabetes and the eye. Available at: https://idf.org/our-activities/care-prevention/eye-health.html; 2020 Accessed 20.08.21.

      As reported by the Vision Loss Expert Group, the prevalence of DR is increasing in many regions globally, with the largest increase in southern sub-Saharan Africa.
      GBD 2019 Blindness and Vision Impairment Collaborators, Vision Loss Expert Group of the Global Burden of Disease Study. Causes of blindness and vision impairment in 2020 and trends over 30 years, and prevalence of avoidable blindness in relation to VISION 2020: the Right to Sight: an analysis for the Global Burden of Disease Study.
      It is estimated that by 2040, 224 million people globally will harbor some form of DR, with vision threatened in 70 million of these people worldwide.
      • Thomas R.L.
      • Halim S.
      • Gurudas S.
      • et al.
      IDF diabetes atlas: a review of studies utilizing retinal photography on the global prevalence of diabetes related retinopathy between 2015 and 2018.
      Although early diagnosis and treatment of DR through screening reduces vision loss by 98%,
      Early Treatment Diabetic Retinopathy Study Research Group
      Early photocoagulation for diabetic retinopathy. ETDRS report number 9.
      ,
      Diabetic Retinopathy Study Research Group
      Photocoagulation treatment of proliferative diabetic retinopathy: clinical application of diabetic retinopathy study (DRS) findings, DRS report number 8.
      low-resource settings such as Rwanda often lack the infrastructure and trained personnel to implement DR screening programs effectively.
      • Poore S.
      • Foster A.
      • Zondervan M.
      • Blanchet K.
      Planning and developing services for diabetic retinopathy in sub-Saharan Africa.
      Furthermore, where screening programs exist in sub-Saharan Africa, many patients identified with referable DR fail to comply with follow-up appointments.
      • Mtuya C.
      • Cleland C.R.
      • Philippin H.
      • et al.
      Reasons for poor follow-up of diabetic retinopathy patients after screening in Tanzania: a cross-sectional study.
      Poor adherence to treatment and follow-up recommendations because of financial barriers, travel time, lack of clarity in the referral process, and uncertainty among patients about the treatability of the disease remain a significant barrier to positive clinical outcomes and preventing vision loss in patients with diabetes.
      • Vengadesan N.
      • Ahmad M.
      • Sindal M.D.
      • Sengupta S.
      Delayed follow-up in patients with diabetic retinopathy in South India: social factors and impact on disease progression.
      Recent advances in computer-based analysis using artificial intelligence (AI) present a promising opportunity to test and refine automatic grading of diabetic retinal images for screening. Using validated AI algorithms instead of scarce trained specialists could potentially increase the efficiency and accessibility of screening programs.
      • Li Z.
      • Keel S.
      • Liu C.
      • et al.
      An automated grading system for detection of vision-threatening referable diabetic retinopathy on the basis of color fundus photographs.
      Systematic reviews of deep learning-based algorithms in DR screening have highlighted such advantages as reduction in demands for manpower, cost of screening, and intragrader and intergrader variability.
      • Nielsen K.B.
      • Lautrup M.L.
      • Andersen J.K.H.
      • et al.
      Deep learning-based algorithms in screening of diabetic retinopathy: a systematic review of diagnostic performance.
      However, evidence from real-life screening programs is limited, and no evidence exists on community acceptance of the use of AI-supported DR screening.
      Studies on the use of AI to screen for DR in Africa report good accuracy of the technology
      • Bellemo V.
      • Lim Z.W.
      • Lim G.
      • et al.
      Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study.
      ,
      • Hansen M.B.
      • Abràmoff M.D.
      • Folk J.C.
      • et al.
      Results of automated retinal image analysis for detection of diabetic retinopathy from the Nakuru study, Kenya.
      ; however, the continent lags in deployment of AI in clinical settings. The present trial, piggybacking on an AI-based DR screening and service-delivery project in Kigali, Rwanda, was designed to assess whether use of Orbis International’s Cybersight AI in diabetes clinics leads to increased patient uptake of DR referral services. Our hypothesis was that adherence to referral services would be higher among patients randomized to receive AI-supported screening with immediate feedback compared with those randomized to receive delayed communication of results until after human grading was completed.

      Methods

      Study Design and Participants

      The Rwanda Artificial Intelligence for Diabetic Retinopathy Screening (RAIDERS) study was an investigator-masked, parallel-group randomized controlled trial enrolling participants at 4 clinical sites in Rwanda. The study was approved by the Rwanda National Health Research Committee (identifier, NHRC/2020/PROT/025) and the Rwanda National Ethics Committee (identifier, 945/RNEC/2020). All participants provided written informed consent before enrollment. The tenets of the Declaration of Helsinki were followed throughout. The trial was registered on the Pan African Clinical Trial Registry (www.pactr.org; registry number, PACTR202101512465690).
      Screening for DR using retinal imaging with AI interpretation was implemented from March 2021 through June 2021 at 4 diabetes clinics in and around Kigali, Rwanda (2 district-level clinics, 1 referral-level clinic, and 1 diabetes association-led clinic). Follow-up at the referral site continued through July 2021. Participants were recruited during routine visits to the diabetologist. The diabetologist or other attending clinician presented the patient information sheet and consent form to all potentially eligible patients before DR screening.
      The study inclusion criteria were diagnosis of type 1 or 2 diabetes, ≥ 18 years of age, provision of informed consent, availability of gradable digital retinal images for ≥ 1 eye, willingness and ability to travel to the designated referral clinic, and lack of any current eye treatment or participation in any ongoing study requiring regular appointments for eye care. Additionally, all participants in both study groups were required to have a positive finding of referable DR or other condition requiring referral for additional investigations according to Cybersight AI. Patients with known DR, currently under the care of an eye doctor, participants who did not provide consent, and those with ungradable images received appropriate clinical care, but were excluded from the study.
      Sample size estimates were based on obtaining data for the primary outcome: uptake of referrals within 30 days of receiving positive screening results. Using an uptake of eye examinations of 35% in the control group and 60% in the treatment group, a power of 90%, and a 2-sided α value of 5%, the estimated target sample size was 79 participants in each group. However, we continued to 275 total participants (137 in each group) to allow us to detect an uptake difference of 20%. With an expected screening positivity rate of 33%, we aimed to enroll 825 participants for the study.

      AI Model

      The model is based on the Inception ResNet version 2 convolutional neural network architecture
      • Szegedy C.
      • Ioffe S.
      • Vanhoucke V.
      • Alemi A.A.
      Inception-v4, inception-ResNet and the impact of residual connections on learning.
      trained to classify fundus photographs into 1 of 5 categories based on the International Clinical Disease Severity Scale for DR (nonproliferative normal, nonproliferative mild, nonproliferative moderate, nonproliferative severe, and proliferative).
      The input to the convolutional neural network is a single (or batch thereof) fundus photograph. Preprocessing includes: (1) removal of black fundus border, if present, and (2) resizing of the image to 448 × 448 pixels with preserved aspect ratio. The output of the model is an L1-normalized vector (i.e., sums to 1), where each element corresponds to 1 of the DR grades. The predicted DR grade is the argmax of the raw output vector, and the referable DR score is the sum of the last 3 elements of the output vectors.
      The model was trained on a total of 90 073 photographs that were quality controlled by ≥ 1 board-certified ophthalmologist. Training was carried out “from scratch” based on randomly initialized weights. To improve generalization to unseen images, data augmentation techniques such as random zoom, flipping, and rotation were applied during training.
      Referable DR performance validation carried out based on a balanced hold-out dataset (200 photographs per DR grade for a total of 1000 photographs) and based on an external benchmark dataset resulted in areas under the receiver operating receiver characteristic curve of 96% and 98.5%, respectively.
      • Rogers T.W.
      • Gonzalez-Bueno J.
      • Garcia Franco R.
      • et al.
      Evaluation of an AI system for the detection of diabetic retinopathy from images captured with a handheld portable fundus camera: the MAILOR AI study.

      Procedures for Imaging and Data Collection

      Participants’ baseline demographic data and clinical characteristics were collected on electronic devices using KoBoToolbox before retinal imaging. After imaging, a questionnaire was administered to all participants inquiring about satisfaction with the screening process and their eye care history and knowledge. Trained personnel captured 2-field (optic disc and center-cantered) digital color, nonstereo, nonmydriatic 45° retinal fundus photographs of each eye (Topcon NW400; Topcon). Retinal images were captured in the JPEG (joint photographic experts group) format, with a dimension of 2592 × 1944 pixels. If image quality was deemed poor because of a small pupillary aperture (< 2.5 mm), the eye was dilated with a single drop of tropicamide 0.5%, and the image was reacquired after 15 minutes.
      After imaging was completed, all images, anonymized with a unique patient registration number, were uploaded to Orbis International’s Cybersight AI. A mobile device or laptop and an internet connection are required to access Cybersight AI, which generates a response regarding the presence or absence of referable DR based on a macula-centered image from each available eye of a participant within 60 seconds. The system automatically confirms that each image contains the correct features and is of sufficient quality for grading. Cybersight AI is available free of charge to eye health professionals in low- and middle-income countries and is accessible on completion of no-charge registration on Orbis International’s telehealth platform, Cybersight. All images also were uploaded to Labelbox (Labelbox, Inc) for grading by a United Kingdom National Health System formally trained retinal specialist.

      Randomization and Masking

      After imaging and interpretation of images by AI, eligible participants whose screening results were positive for referral by AI were randomly assigned (1:1) to either grading by AI with immediate feedback (intervention) or grading by human graders with communication of need for referral only after human grading was completed in 3 to 5 days (control). The study group was assigned for each participant by having them flip a coin that read “AI” on one side and “Human” on the other side. A few participants reluctant to toss a coin were randomized using sealed, opaque envelopes that had either “AI” or “Human” written on a sheet. The decision according to the AI system was used to determine referral for both study groups to guarantee that they would be similar at baseline, although only participants randomized to the intervention (AI) group received their reports immediately.
      Clinic staff capturing and uploading retinal images, and those collecting outcome data at the referral site (such as receiving clerks entering attendance data), were masked to participant group assignment. All images from potential participants, regardless of the AI grade determining enrollment in the study, were graded by human experts masked to the AI grade, and thus participant inclusion or group assignment. For practical reasons, study participants, the study coordinator, and study personnel responsible for interviews and randomization were not masked; however, participants remained unaware of the study hypothesis and primary outcome.
      Participants randomized to the intervention (AI) group were made aware that their screening report was automatically generated by the AI platform. No additional education on the AI system was provided. However, the intervention group was aware that their images would also be reviewed by human graders. Intervention participants received a report that included their fundus images and was color coded for severity of the DR grade (green, no DR; yellow, mild DR; orange, moderate DR; and red, vision-threatening DR). At this time, intervention participants were informed that referral to a secondary clinic for further ocular examination was required. Additional referral criteria included a cup-to-disc ratio of > 0.7 or any macular anomaly.
      Participants in the control (human grading) group were unaware of the AI report, but were aware that they would be contacted about whether to follow-up at the eye clinic through short message service (SMS) and also through a phone call from the health worker who attended them, after the human grading report was completed in 3 to 5 days. Only after screeners had received human grading reports were control participants informed that they needed to visit the referral clinic. When AI findings had been positive (patient thus recruited into trial) and human grading results were negative, the patient remained in the trial and was informed of the status after the full follow-up examination.
      Participants in both study groups received clear instructions about the follow-up process, including the location of the eye clinic and information on reimbursement of travel costs and insurance copayments. The referral site was a secondary-level clinic with an ophthalmologist skilled in the management of DR. At the time of receiving the report on the examination (immediately for the intervention group, or after 3 to 5 days for the control group), participants were informed that they could report to the eye clinic on any working day within the next 30 days.

      Assessment of Outcomes

      The main study outcome is the difference between study groups in the rate of presentation for recommended referral services within 30 days of being informed of the need for a referral visit. Attendance for eye examination at the eye clinic was recorded on an outcome form by an independent research assistant who was masked to participant study group assignment. The project manager received the completed outcome forms and linked the data to the participant database for each participant. Nonattendance was defined as failure to attend the referral clinic on any occasion within 30 days of being recommended for follow-up. Participants in the control group who could not be successfully contacted by SMS or telephone (n = 5) were still included in the denominator as requiring referral.

      Statistical Analysis

      Patients with nondeferrable outcomes on images according to the AI were not enrolled in the trial and were excluded from the analyses. Demographic and clinical characteristics are presented stratified by study group according to the principle of intention to treat. Frequencies and percentages for categorical variables and means and standard deviations for continuous variables are presented, stratified by study group. Responses of “I don’t know” regarding awareness of eye care knowledge and beliefs are categorized as negative or incorrect responses. Rates of attendance at the referral clinic visit (main outcome) are described as unadjusted percentages, comparing the 2 study groups using the chi-square test. Univariate and multivariate logistic regression modeling was used to compare the main outcome between study groups and other potential predictors. Variables that were significant in the univariate models were included in the multivariate model if they remained significant (P < 0.05).

      Role of the Funding Source

      The project was supported by Orbis International and the Association for Research in Vision and Ophthalmology (Roche Award). The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.

      Results

      Of the 827 clinic patients with diabetes who gave consent and were screened for DR, 4 did not meet inclusion criteria, specifically the ability to travel. A total of 275 participants (33.4%) met the referral criteria on AI screening and were randomized for inclusion in the trial, with 136 (49.5%) allocated to the intervention group and 139 (50.5%) allocated to the control group. All participants (100%) in each study group received the allocated intervention, and data for the main outcome was available for all (100%) participants (Fig 1).
      Figure thumbnail gr1
      Figure 1Flowchart showing Rwanda Artificial Intelligence for Diabetic Retinopathy Screening trial enrollment. AI = artificial intelligence; DR = diabetic retinopathy.
      Baseline demographic data (Table 1) did not differ significantly between study groups. Satisfaction with the screening process was very high in both groups (intervention, 100%; control, 99.3%). A total of 5 participants (3.60%) in the control group could not be contacted by SMS or telephone, but were retained in the analysis as specified pre hoc in the registered protocol. None returned for follow-up.
      Table 1Baseline Demographic and Clinical Characteristics by Study Group
      VariableIntervention Group (n = 136)Control Group (n = 139)
      Demographic
       Mean age (yrs)50.1 ± 16.051.3 ± 15.9
       Female sex79 (58.1)81 (58.3)
      Educational level
       None25 (18.4)22 (15.8)
       Primary41 (30.2)49 (35.2)
       Secondary47 (34.6)48 (34.5)
       Tertiary23 (16.9)20 (14.4)
      Socioeconomic status
      Status based on official Ubudehe classification that exists for all Rwandans (https://rwandapedia.rw/hgs/ubudehe/poverty-level-categories) and that was reviewed in 2020. Category A is highest, categories B and C are combined into medium, and categories D and E are the lowest. No statistically significant differences exist between the two study groups.
       Highest18 (13.2)14 (10.1)
       Medium99 (72.8)113 (81.3)
       Lowest4 (2.9)4 (2.9)
       Unknown15 (11.0)8 (5.8)
      Health insurance
       None6 (4.4)6 (4.3)
       Public117 (86.0)124 (89.2)
       Private8 (5.9)4 (2.9)
       Other5 (3.7)5 (3.6)
      Occupation
       Professional8 (5.9)10 (7.2)
       Skilled work31 (22.8)27 (19.4)
       Unskilled work17 (12.5)18 (13.0)
       Unemployed71 (52.2)80 (57.6)
       Retired/pensioner9 (6.6)4 (2.9)
      Diabetes status
       Type of diabetes
      163 (46.3)56 (40.3)
      271 (52.2)80 (57.6)
      Unknown2 (1.5)3 (2.2)
       Duration (yrs)
      <547 (34.6)36 (25.9)
      5–1043 (31.6)45 (32.4)
      >1046 (33.8)58 (41.7)
       Blood glucose (mg/dl)8.97 ± 3.69 (n = 123)10.3 ± 5.01 (n = 120)
      Eye care history and knowledge
       Patient reports dilated eye examination in past year14 (10.3)23 (16.6)
       Aware diabetes can cause eye problems121 (89.0)125 (89.9)
       Personally knows a blind person64 (47.1)64 (46.4)
       Worried about losing sight110 (80.9)112 (80.6)
      High satisfaction with screening processes136 (100)137 (99.3)
      Mean distance of home from referral site (km)17.2 ± 20.115.8 ± 18.4
      Residence
       Urban79 (58.1)85 (61.2)
       Rural57 (41.9)54 (38.8)
      Data are presented as no. (%) or mean±standard deviation.
      Status based on official Ubudehe classification that exists for all Rwandans (https://rwandapedia.rw/hgs/ubudehe/poverty-level-categories) and that was reviewed in 2020. Category A is highest, categories B and C are combined into medium, and categories D and E are the lowest. No statistically significant differences exist between the two study groups.
      Adherence with recommended referral examination was significantly higher in the intervention group (70/136 [51.5%]) versus the control group (55/139 [39.6%]; P = 0.048) in unadjusted analyses (Table 2), representing a 30.1% increase. When adjusted for age, sex, and urban or rural residence, membership in the intervention was significantly associated with acceptance of recommended referral (odds ratio, 1.74; 95% confidence interval, 1.05–2.88; P = 0.031; Table 3). Participants in the AI group also sought treatment at the referral clinic much sooner after receiving referral advice than those in the control group: AI group, 6.6 ± 7.4 days (median, 4.0 days; range, 0–30 days) versus control group, 9.6 ± 5.1 days (median, 8.0 days; range, 3–28 days; P < 0.0001, Wilcoxon rank-sum test).
      Table 2Unadjusted Comparison of Primary Outcome between the Intervention and Control Groups
      Referral OutcomeIntervention Group (n = 136)Control Group (n = 139)P Value
      Adhered with referral, no. (%)70 (51.5)55 (39.6)0.048
      Did not adhere with referral, no. (%)66 (48.5)84 (60.4)
      Table 3Univariate and Multivariate Comparison of Primary Outcome (Referral Adherence)
      VariablesOdds Ratio95% Confidence IntervalP Value
      Univariate
      Intervention group1.621.00–2.610.048
      Age, years1.031.01–1.050.0002
      Male sex1.500.93–2.430.099
      Educational level
       NoneReference
       Primary1.410.69–2.870.342
       Secondary0.940.46–1.910.864
       Tertiary1.170.51–2.700.706
      Socioeconomic status
       HighestReference
       Medium1.600·75–3.450.225
       Lowest0.240.03–2.180.204
       Unknown0.890.29–2.720.836
      Health insurance
       NoneReference
       Public1.760.52–6.020.364
       Private1.430.27–7.520.674
       Other0.860.14–5.230.867
      Occupation
       Professional0.430.15–1.280.129
       Skilled work1.130.62–2.060.699
       Unskilled work0.590.27–1.270.175
       UnemployedReference
       Retired/pensioner1.800.56–5.760.320
      Type 2 diabetes (type 1 is reference)2.281.39–3.760.001
      Duration of diabetes (yrs)
       <5Reference
       5–101.100.60–2.010.769
       >101.500.84–2.680.174
      Blood glucose (mg/dl)1.000.94–1.060.951
      Patient reports dilated eye examination in past year0.900.45–1.810.772
      Aware diabetes can cause eye problems1.200.55–2.630.642
      Personally knows a blind person0.990.62–1.590.965
      Worried about losing sight0.920.50–1.670.780
      High satisfaction with screening processesNot calculable
      Not calculable because of zero cell size in the denominator.
      Not calculable
      Not calculable because of zero cell size in the denominator.
      0.986
      Distance of home from referral site (km)1.011.00–1.030.081
      Rural vs. urban residence1.681.04–2.740.036
      Multivariate
      Intervention group1.731.04–2.870.034
      Male sex2.081.22–3.540.007
      Age (yrs)1>041.02–1.05< 0.0001
      Rural residence1.771.05–2.990.033
      Not calculable because of zero cell size in the denominator.

      Discussion

      Screening programs are a proven, cost-effective model for preventing serious complications resulting from DR,
      • Javitt J.C.
      • Aiello L.P.
      Cost-effectiveness of detecting and treating diabetic retinopathy.
      • Pieczynski J.
      • Grzybowski A.
      Review of diabetic retinopathy screening methods and programmes adopted in different parts of the world.
      • Wong T.Y.
      • Sabanayagam C.
      Strategies to tackle the global burden of diabetic retinopathy: from epidemiology to artificial intelligence.
      but they are dependent on adherence with referral services to succeed in preventing negative clinical outcomes. Previous studies have reported successful DR screening programs based in primary care and diabetes clinics,
      • Liu J.
      • Gibson E.
      • Ramchal S.
      • et al.
      Diabetic retinopathy screening with automated retinal image analysis in a primary care setting improves adherence to ophthalmic care.
      • Mumba M.
      • Hall A.
      • Lewallen S.
      Compliance with eye screening examinations among diabetic patients at a Tanzanian referral hospital.
      • Zhang Y.
      • Shi J.
      • Peng Y.
      • et al.
      Artificial intelligence-enabled screening for diabetic retinopathy: a real-world, multicenter and prospective study.
      • Scheetz J.
      • Koca D.
      • McGuinness M.
      • et al.
      Real-world artificial intelligence-based opportunistic screening for diabetic retinopathy in endocrinology and indigenous healthcare settings in Australia.
      and Bellemo et al
      • Bellemo V.
      • Lim Z.W.
      • Lim G.
      • et al.
      Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study.
      demonstrated the validity of using AI for DR screening in an underresourced African population. Similar to Liu et al,
      • Liu J.
      • Gibson E.
      • Ramchal S.
      • et al.
      Diabetic retinopathy screening with automated retinal image analysis in a primary care setting improves adherence to ophthalmic care.
      the current study focused on adherence with screening referral as a crucial step in the practical application of AI-supported DR screening in low-resource settings, yet the RAIDERS study strengthens the evidence because of its randomized clinical trial design. The RAIDERS trial found increased adherence to DR referral services among participants randomized to receive immediate feedback on referral status based on AI-supported screening compared with those randomized to receiving referral advice only after the human grading report was completed.
      Data on adherence with follow-up after DR screening in low-resource countries are limited. The adherence rate for the intervention (AI) group in this study (51.5%) is higher than rates reported in traditional screening programs not delivering immediate feedback on need for referral, including a study in the neighboring country of Tanzania (25%),
      • Mtuya C.
      • Cleland C.R.
      • Philippin H.
      • et al.
      Reasons for poor follow-up of diabetic retinopathy patients after screening in Tanzania: a cross-sectional study.
      and also in more developed countries, where follow-up rates as low as 45.2%
      • Watane A.
      • Kalavar M.
      • Cavuoto K.M.
      • Sridhar J.
      Factors Associated with follow-up non-compliance in patients presenting to an emergency department with non-proliferative diabetic retinopathy.
      and 36%
      • Pernic A.
      • Kaehr M.M.
      • Doerr E.D.
      • et al.
      Analysis of compliance with recommended follow-up for diabetic retinopathy in a county hospital population.
      have been reported. Artificial intelligence-supported screening provides an opportunity for immediate counselling and eye health education for those requiring referral, potentially contributing to increased adherence. Other studies in low-resource settings have reported improved adherence with referral care in response to eye health education.
      • Keenum Z.
      • McGwin G.
      • Witherspoon C.D.
      • et al.
      Patients’ adherence to recommended follow-up eye care after diabetic retinopathy screening in a publicly funded county clinic and factors associated with follow-up eye care use.
      ,
      • Khair Z.
      • Rahman M.M.
      • Kazawa K.
      • et al.
      Health education improves referral compliance of persons with probable diabetic retinopathy: a randomized controlled trial.
      Provision of an instant report that includes images of the retina may support acceptance of the recommended referral, especially for those who are asymptomatic. Furthermore, in this model, participants who wished to visit the ophthalmologist on the same day as screening could do so, potentially minimizing travel and accounting for the quicker uptake of referral seen in this study, although additional research on factors contributing to increased adherence is needed. The study by Watane et al
      • Watane A.
      • Kalavar M.
      • Cavuoto K.M.
      • Sridhar J.
      Factors Associated with follow-up non-compliance in patients presenting to an emergency department with non-proliferative diabetic retinopathy.
      found that longer recommended intervals for follow-up led to increased odds of follow-up nonadherence. We hypothesized that an important reason for lower acceptance of care in the control group would be an inability to contact these participants after they left the clinic, but in fact, nonadherence for this reason was not common. Follow-up rates in the control group (39.6%) were higher than those in the intervention group in Tanzania, potentially because of high insurance coverage rates in Rwanda, which reduced barriers of cost, and the impact of phone reminders to relay the results of the DR screening. Such contact with SMS reminders
      • Chen T.
      • Zhu W.
      • Tang B.
      • et al.
      A mobile phone informational reminder to improve eyecare adherence among diabetic patient in rural China: a randomized controlled trial.
      has been shown to increase adherence with recommended eye care.
      Although an increase in the uptake of referral services was observed in the intervention group compared with the control group in the current study, close to half of those undergoing AI screening did not comply with referral care within 30 days. Clearly, other interventions in combination with AI-supported screening are needed to increase adherence further.
      Strengths of the study include results that are broadly applicable across AI-supported DR screening models, regardless of the AI platform used. The trial was a randomized design with good fidelity to protocol and high follow-up rates. Additionally, drawing participants from different types of facilities provided data representative of the broader Rwandan health system. Furthermore, the study provides evidence on the practical application of AI-supported DR screening in low-resource settings, specifically in Africa, where few trial data on the programmatic impact of AI exist. Key factors that should be considered when thinking about a similar approach in other locations include support from the diabetes care providers to provide time and space for eye screening, the availability and accessibility of an accurate AI system, using a high-quality nonmydriatic camera for imaging acquisition, and ensuring confidence in the system by those presenting it to patients. Acknowledging limitations, the study included a relatively higher proportion of patients from urban areas (60%), meaning that the application of the results to rural settings must be made with caution. Accuracy of the AI platform used has not been validated in a peer-reviewed publication for those of African descent, but the accuracy of the referral was not relevant to the main trial outcome, and overread precautions were implemented. Finally, it was impractical to mask participants, but placebo effects are not of particular concern for the main outcome of this study.
      In conclusion, this study demonstrated the potential of AI-supported DR screening to deliver increased uptake of referral services. Results of the RAIDERS trial provide evidence for the integration of AI for DR screening as part of a sustainable national eye care program to prevent DR-related blindness in sub-Saharan Africa. Additional research on methods to further increase adherence with referral services is needed, because early diagnosis and treatment of DR is highly effective in preventing vision loss.
      Early Treatment Diabetic Retinopathy Study Research Group
      Early photocoagulation for diabetic retinopathy. ETDRS report number 9.
      ,
      Diabetic Retinopathy Study Research Group
      Photocoagulation treatment of proliferative diabetic retinopathy: clinical application of diabetic retinopathy study (DRS) findings, DRS report number 8.
      Further research on the use of AI-supported DR screening in low- to middle-income countries is also necessary to better understand how to integrate this technology into efficient DR care systems in low-resource settings.

      References

      1. International Diabetes Federation. International Diabetes Federation diabetes atlas, 9th ed. Available at: https://www.diabetesatlas.org; 2019 Accessed 20.08.21.

      2. International Diabetes Federation. Diabetes and the eye. Available at: https://idf.org/our-activities/care-prevention/eye-health.html; 2020 Accessed 20.08.21.

      3. GBD 2019 Blindness and Vision Impairment Collaborators, Vision Loss Expert Group of the Global Burden of Disease Study. Causes of blindness and vision impairment in 2020 and trends over 30 years, and prevalence of avoidable blindness in relation to VISION 2020: the Right to Sight: an analysis for the Global Burden of Disease Study.
        Lancet Glob Health. 2021; 9 (erratum in: Lancet Glob Health. 2021;9(2):e144-e160): e144-e160
        • Thomas R.L.
        • Halim S.
        • Gurudas S.
        • et al.
        IDF diabetes atlas: a review of studies utilizing retinal photography on the global prevalence of diabetes related retinopathy between 2015 and 2018.
        Diabetes Res Clin Pract. 2019; 157107840
        • Early Treatment Diabetic Retinopathy Study Research Group
        Early photocoagulation for diabetic retinopathy. ETDRS report number 9.
        Ophthalmology. 1991; 98: 766-785
        • Diabetic Retinopathy Study Research Group
        Photocoagulation treatment of proliferative diabetic retinopathy: clinical application of diabetic retinopathy study (DRS) findings, DRS report number 8.
        Ophthalmology. 1981; 88: 583-600
        • Poore S.
        • Foster A.
        • Zondervan M.
        • Blanchet K.
        Planning and developing services for diabetic retinopathy in sub-Saharan Africa.
        Int J Health Policy Manag. 2014; 4: 19-28
        • Mtuya C.
        • Cleland C.R.
        • Philippin H.
        • et al.
        Reasons for poor follow-up of diabetic retinopathy patients after screening in Tanzania: a cross-sectional study.
        BMC Ophthalmol. 2016; 16: 115https://doi.org/10.1186/s12886-016-0288-z
        • Vengadesan N.
        • Ahmad M.
        • Sindal M.D.
        • Sengupta S.
        Delayed follow-up in patients with diabetic retinopathy in South India: social factors and impact on disease progression.
        Indian J Ophthalmol. 2017; 65: 376-384
        • Li Z.
        • Keel S.
        • Liu C.
        • et al.
        An automated grading system for detection of vision-threatening referable diabetic retinopathy on the basis of color fundus photographs.
        Diabetes Care. 2018; 41: 2509-2516
        • Nielsen K.B.
        • Lautrup M.L.
        • Andersen J.K.H.
        • et al.
        Deep learning-based algorithms in screening of diabetic retinopathy: a systematic review of diagnostic performance.
        Ophthalmol Retina. 2019; 3: 294-304
        • Bellemo V.
        • Lim Z.W.
        • Lim G.
        • et al.
        Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study.
        Lancet Digit Health. 2019; 1: e35-e44
        • Hansen M.B.
        • Abràmoff M.D.
        • Folk J.C.
        • et al.
        Results of automated retinal image analysis for detection of diabetic retinopathy from the Nakuru study, Kenya.
        PLoS One. 2015; 10e0139148
        • Szegedy C.
        • Ioffe S.
        • Vanhoucke V.
        • Alemi A.A.
        Inception-v4, inception-ResNet and the impact of residual connections on learning.
        in: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI’17). 1st. 31. AAAI Press, 2017: 4278-4284 (1st ed. AAAI Press; 2017;31:4278--4284. https://ojs.aaai.org/index.php/AAAI/article/view/11231. Accessed August 20, 2021.)
        • Rogers T.W.
        • Gonzalez-Bueno J.
        • Garcia Franco R.
        • et al.
        Evaluation of an AI system for the detection of diabetic retinopathy from images captured with a handheld portable fundus camera: the MAILOR AI study.
        Eye. 2020; 35: 632-638
        • Javitt J.C.
        • Aiello L.P.
        Cost-effectiveness of detecting and treating diabetic retinopathy.
        Ann Intern Med. 1996; 124: 164-169
        • Pieczynski J.
        • Grzybowski A.
        Review of diabetic retinopathy screening methods and programmes adopted in different parts of the world.
        Eur Oph Rev. 2015; 9: 49-55
        • Wong T.Y.
        • Sabanayagam C.
        Strategies to tackle the global burden of diabetic retinopathy: from epidemiology to artificial intelligence.
        Ophthalmologica. 2020; 243: 9-20
        • Liu J.
        • Gibson E.
        • Ramchal S.
        • et al.
        Diabetic retinopathy screening with automated retinal image analysis in a primary care setting improves adherence to ophthalmic care.
        Ophthalmol Retina. 2021; 5: 71-77
        • Mumba M.
        • Hall A.
        • Lewallen S.
        Compliance with eye screening examinations among diabetic patients at a Tanzanian referral hospital.
        Ophthalmic Epidemiol. 2007; 14: 306-310
        • Zhang Y.
        • Shi J.
        • Peng Y.
        • et al.
        Artificial intelligence-enabled screening for diabetic retinopathy: a real-world, multicenter and prospective study.
        BMJ Open Diabetes Res Care. 2020; 8e001596
        • Scheetz J.
        • Koca D.
        • McGuinness M.
        • et al.
        Real-world artificial intelligence-based opportunistic screening for diabetic retinopathy in endocrinology and indigenous healthcare settings in Australia.
        Sci Rep. 2021; 1115808
        • Watane A.
        • Kalavar M.
        • Cavuoto K.M.
        • Sridhar J.
        Factors Associated with follow-up non-compliance in patients presenting to an emergency department with non-proliferative diabetic retinopathy.
        Invest Ophthalmol Vis Sci. 2020; 61: 3831
        • Pernic A.
        • Kaehr M.M.
        • Doerr E.D.
        • et al.
        Analysis of compliance with recommended follow-up for diabetic retinopathy in a county hospital population.
        Invest Ophthalmol Vis Sci. 2014; 55: 2290
        • Keenum Z.
        • McGwin G.
        • Witherspoon C.D.
        • et al.
        Patients’ adherence to recommended follow-up eye care after diabetic retinopathy screening in a publicly funded county clinic and factors associated with follow-up eye care use.
        JAMA Ophthalmol. 2016; 134: 1221-1228
        • Khair Z.
        • Rahman M.M.
        • Kazawa K.
        • et al.
        Health education improves referral compliance of persons with probable diabetic retinopathy: a randomized controlled trial.
        PLoS One. 2020; 15e0242047
        • Chen T.
        • Zhu W.
        • Tang B.
        • et al.
        A mobile phone informational reminder to improve eyecare adherence among diabetic patient in rural China: a randomized controlled trial.
        Am J Ophthalmol. 2018; 194: 54-62