Comparison of Methods of Clinical Trial Emulation Utilizing Data From the Comparison of AMD Treatment Trial (CATT) and the IRIS® Registry

Purpose We used exact matching and inverse propensity score weighting (IPSW) using real-world data (RWD) from the American Academy of Ophthalmology IRIS® Registry (Intelligent Research in Sight) to emulate the 2 pro re nata (prn) treatment arms from the Comparison of AMD Treatment Trial (CATT) and to compare the outcomes of the RWD arms to the 2 monthly treatment arms from the clinical trial. Design Retrospective cohort study utilizing deidentified electronic health record registry data and patient-level deidentified clinical trial data. Subjects All treatment-naive patient eyes with neovascular age-related macular degeneration treated with ranibizumab or bevacizumab only for 1 year from either the CATT or the IRIS Registry. Methods Patients were identified in the IRIS Registry between October 1, 2015 and December 31, 2019. After all nonimaging-based inclusion and exclusion criteria from the CATT were applied, patient eyes receiving bevacizumab or ranibizumab only on a prn basis were identified as the eligible cohort. Exact matching and ISPW was applied based on age, gender, and baseline visual acuity. Main Outcome Measures Mean change in visual acuity, in approximated ETDRS letters, between baseline and 1 year for the IRIS Registry prn treatment arms generated by exact matching and IPSW. Results We identified 427 eyes treated with ranibizumab prn and 771 eyes treated with bevacizumab prn. Using exact matching, 98% (n = 281) of CATT patient eyes in the bevacizumab monthly treatment arm and 87% (n = 261) of CATT patient eyes in the ranibizumab monthly treatment arm were matched to a patient eye in the IRIS Registry. For the ranibizumab prn treatment arm, patient eyes generated using exact matching gained 1.9 letters and those generated using IPSW gained 2.8 letters (exact matching: 1.9 letters ± 14.0 vs. IPSW: 2.8 letters ± 15.0 letters, P = 0.43). For the bevacizumab prn treatment arm, patient eyes generated using exact matching gained 2.4 letters and those generated using IPSW gained 2.1 letters (exact matching: 2.4 letters ± 15.4 vs. IPSW: 2.1 letters ± 16.0 letters, P = 0.79). Conclusions Both exact matching and IPSW produced similar results in emulating the prn treatment arms of the CATT using IRIS Registry data and patient-level clinical trial data. Similar to prior real-world studies, the clinical outcomes were significantly worse in the IRIS Registry treatment arms compared with the clinical trial. Financial Disclosure(s) Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Comparison of Age-Related Macular Degeneration Treatment Trials (CATT), a National Eye Institute-funded randomized controlled trial investigating the efficacy of ranibizumab versus bevacizumab treatment for neovascular age-related macular degeneration (nAMD) administered via 2 dosing schedules (monthly and pro re nata [prn]), was selected for emulation. 8The scale and breadth of the American Academy of Ophthalmology IRIS Ò Registry (Intelligent Research in Sight), which is the world's largest ophthalmology EHR registry and contains >6.4 million age-related macular degeneration patients, enabled its use as a source of RWD.
This study aimed to compare 2 methodologies, exact matching and inverse propensity score weighting (IPSW), using RWD from the IRIS Registry to emulate the 2 prn treatment arms from CATT and to compare the outcomes of these real-world arms to the 2 monthly treatment arms from the clinical trial.

Methods
Deidentified data from the IRIS Registry, which, as of July 2022, contained >75 million patients, were used for this analysis.This study complies with the tenets of the Declaration of Helsinki.Data stored within the IRIS Registry were deidentified and have been qualified as Health Insurance Portability and Accountability Act compliant.The study was reviewed and deemed exempt under 45 CFR x 46.104(d)(4) by the WIRB-Copernicus Group Institutional Review Board (Puyallup, WA).No patient consent was required because this study was certified as conducted on anonymized, deidentified data used for research purposes by Verana Health.
All treatment-naive patient eyes with nAMD in the IRIS Registry who received their first anti-VEGF injection between October 1, 2015 and December 31, 2019 were eligible for study inclusion based on International Classification of Diseases, 10 th Edition and Current Procedural Terminology coding.Treatment-naive was defined as no documentation of anti-VEGF injections in structured EHR data (Current Procedural Terminology and Healthcare Common Procedure Coding System codes) for 1 year prior to the index date, or date of first injection.
Additional inclusion and exclusion criteria were applied to reflect (1) all nonimaging based eligibility criteria from the CATT and (2) the treatment pattern required in the CATT of prn treatment with either bevacizumab or ranbizumab for 1 year. 8These are listed in detail in Table 1.Specifically, patient eyes that received bevacizumab only or ranibizumab only for 1 year after the initial index date were included.If a patient was receiving anti-VEGF injections in both eyes during the study period, the eye with the earlier index date was included in the study.As needed or prn treatment was defined as a treatment regimen requiring a clinical encounter with a retina specialist every 4 to 6 weeks for 1 year after the index date and !1 clinical encounter with a retina specialist that was not associated with an anti-VEGF intravitreal injection.
Patient demographic data, including age, gender, and race, and clinical information, including visual acuity (VA), number of injections, and incidence of endophthalmitis, was ascertained from IRIS Registry data for the real-world cohorts.Visual acuity was determined based on the best documented Snellen VA of a study eye for the clinical encounter.Best documented Snellen VA was then converted to an approximated ETDRS letter equivalent for statistical analysis using the formula 85 þ 50 Â log (Snellen fraction) rounded to the nearest letter; this formula has been described in more detail previously. 9ter the bevacizumab and ranibizumab cohorts were determined, 2 methodologies were used to generate real-world treatment arms to compare to the monthly ranibizumab only and the monthly bevacizumab only treatment arms from the CATT using deidentified patient-level clinical trial data, which is publicly available at https://hyperprod.cceb.med.upenn.edu/catt/catt_index.php.Figure 1 provides a schematic of the treatment arms generated from IRIS Registry and the outcomes evaluated.Exact matching was used to match a patient eye 1:1 from the IRIS Registry to a patient eye in the CATT on the basis of age, gender, and baseline VA at the time of initial anti-VEGF treatment, similar to what was done in a prior United Kingdom-based paper utilizing exact matching to create a synthetic control arm for an nAMD clinical trial. 7Age was matched within 5 years of the clinical trial participant age at baseline visit, with the exception of patients !90 years of age, who were matched into this broader category due to deidentification purposes.Visual acuity was matched within 5 letters of baseline VA between CATT patients and IRIS Registry patients.
Inverse propensity score weighting was also used to generate a real-world prn cohort for bevacizumab and ranibizumab.A logistic regression model was used to predict probability of exposure to either bevacizumab or ranibizumab and either prn or monthly treatment based on age, gender, and baseline VA.These results were then used to weight observations in the IRIS Registry prn bevacizumab and ranibizumab treatment arms.
The main nonimaging based primary and secondary outcomes were evaluated for the real-world cohorts.Specifically, the primary outcome that was evaluated for the IRIS Registry prn treatment arms was the mean change in VA, in approximated ETDRS letters, between baseline and 1 year.The secondary outcomes evaluated were proportion of patient eyes with a change in VA of !15 letters at 1 year, number of injections at 1 year, annual drug cost (assuming $2000 per dose for ranibizumab and $50 per dose for bevacizumab), and incidence of ocular adverse events.
Clinical outcomes were compared between the exact matching and IPSW prn cohorts for bevacizumab and ranibizumab respectively.Additionally the prn cohorts were compared with the respective monthly cohorts from the CATT.All analyses were conducted using Python 3.7 and Apache Spark 2.4.5 (Apache Software Foundation).P-values were calculated using an unpaired t-test, and a P-value <0.05 was considered statistically significant.

Results
Overall, 435 310 eyes of 376 062 patients with nAMD who initiated anti-VEGF treatment with ranibizumab or bevacizumab were identified during the study period.After nonimaging-based inclusion and exclusion criteria, and required treatment regimen criteria were applied, 427 eyes treated with ranibizumab prn and 771 eyes treated with bevacizumab prn were identified.The details of cohort attrition are provided in Table 1.Baseline characteristics, including age, gender, race, and baseline VA, which are described in Table 2, were comparable between the CATT and eligible IRIS Registry patients.
After exact matching, the CATT bevacizumab monthly treatment arm was comprised of 286 patient eyes, and the ranibizumab monthly treatment arm was comprised of 301 patient eyes. 8When matched on the basis of age, gender, and baseline VA, 98% (n ¼ 281) of CATT patient eyes in the bevacizumab monthly treatment arm and 87% (n ¼ 261) of CATT patient eyes in the ranibizumab monthly treatment arm were able to be exact matched to a patient eye in the IRIS Registry, respectively.Inverse propensity score weighting was then performed on the basis of age, gender, and baseline VA, to provide weights for each of the patient eyes in the IRIS Registry that were identified to meet the clinical trial eligibility criteria and received bevacizumab prn (n ¼ 771) or ranibizumab prn (n ¼ 427) for 1 year.Baseline characteristics for the exact matching and IPSW arms, including age, gender, race, and baseline VA, are described in Table 3.

PRN vs. Monthly Treatment Arms
Clinical outcomes were then evaluated and are described in Table 4 for the CATT monthly and prn ranibizumab and bevacizumab treatment arms and for the IRIS Registry prn treatment arms generated by exact matching and IPSW.Overall, fewer intravitreal injections were performed in the prn treatment arms from the IRIS Registry compared with the monthly treatment arms from the CATT for both bevacizumab and ranibizumab.Additionally, VA  The number of injections in the ranibizumab prn treatment arms generated using the 2 methodologies was exactly the same, 9.0 injections, and the number of injections in the bevacizumab prn treatment arms generated using exact matching and IPSW was 7.9 and 7.7 injections, respectively.

Discussion
Prior studies evaluating the role of RWD to emulate clinical trial arms in ophthalmology have been limited. 7,10The purpose of this study was to compare 2 methodologies, exact matching and IPSW, using RWD from the IRIS Registry to emulate the 2 prn treatment arms from the CATT and to compare the outcomes of these real-world arms to the 2 monthly treatment arms from the clinical trial.Our study used approximated ETDRS letters as its primary clinical outcome and also evaluated injection frequency as a secondary outcome.This study demonstrated that the 2 methodologies, exact matchingandIPSW,generatedtreatment arms with verysimilar clinical outcomes, including change in VA at 1 year and number ofinjectionsin1 year.Additionally, the prn treatment armsfrom the IRIS Registry had worse outcomes with respect to VA gains compared with the monthly treatment arms from the CATT.
There are multiple possible reasons why the 2 methodologies produced such similar results.For one, both the inclusion and exclusion criteria and the treatment regimen requirements were so specific that they yielded a cohort of patient eyes that were very similar due to adjustments made by restriction.Of the 1.6 million treatment-naive patient eyes treated with bevacizumab or ranibizumab in the IRIS Registry, <0.1% met all the nonimaging-based inclusion and exclusion criteria and followed the appropriate treatment regimen.This subgroup of patient eyes was comparable in age, gender, and baseline VA to the CATT monthly treatment arms, suggesting that IPSW may not have adjusted this overall cohort significantly.Because the overall cohort was small, over one-half of the ranibizumab-treated eyes and more than one-third of bevacizumab-treated eyes were included in the exact matching cohort and may partially explain why similar results were observed.Additionally, the scale of the IRIS Registry allowed for the use of exact matching.Other methodologies may be required for instances where the size of a source population for a study is limited.This study also demonstrated that clinical outcomes in the IRIS Registry prn treatment arms were significantly worse than the treatment outcomes in the CATT monthly treatment outcomes.Specifically, patient eyes in the CATT monthly treatment arms for ranibizumab and bevacizumab gained 8.5 and 8.0 letters respectively at 1 year compared with between 2 and 3 letters gained in the IRIS Registry prn treatment arms.Rather than suggest that prn treatment is inferior to monthly treatment, which would be contrary to the CATT results, 8 this validates the results of several prior RWD studies showing that real-world patient outcomes are significantly worse than clinical trial patient outcomes.11e16 Pro re nata treatment in the real-world is likely disrupted by issues, such as insurance coverage and inadvertent lapses in follow up, that are not especially problematic in clinical trials.This is potentially due to other unmeasured or unobserved factors that resulted in continued differences between the trial participants and the RWD cohorts.
Notably, outcomes for patient eyes in the IRIS Registry prn treatment arms were worse than the respective prn treatment arms in the CATT.The patients in the IRIS Registry ranibizumab prn treatment arm in particular who were being seen monthly and treated prn were injected more frequently than the prn treatment arm in the actual CATT trial.This suggests they may have been more refractory to treatment and is also reflected in the worse VA gains seen in the IRIS Registry prn treatment arms despite the same or more frequent treatment.Based on these study results, future trial emulation studies may be better suited to clinical trials that do not have a prn treatment arm since true prn treatment is difficult to ascertain from structured EHR data alone.
There are limited examples of similar studies in the retina literature. 7One prior study in the United Kingdom created a synthetic aflibercept treatment arm using data from the United Kingdom Age-Related Macular Degeneration EMR Users Group to compare to the bevacizumab clinical trial treatment arm for the ABC (Bevacizumab for neovascular age-related macular degeneration) trial. 7,17his study also used exact matching and IPSW considering age, gender, and baseline VA and included only patient eyes receiving standard-of-care treatment.Similar results were noted in this prior study.Specifically, IPSW and exact matching yielded equivocal results in generating a synthetic aflibercept control arm.
There are several strengths to this study.The IRIS Registry is the world's largest single specialty EHR Registry with >6 million nAMD patients. 18This allowed very high matching rates for the bevacizumab and ranibizumab prn treatment arms despite very specific inclusion and exclusion criteria, as well as less commonly used treatment regimens during the study period.Additionally, this proof-of-concept study illustrated a novel application for integrating RWD and clinical trial data using deidentified patient-level clinical trial data, which is unique in ophthalmology.Furthermore, the focus on clinical outcomes and adverse events that are routinely captured in the EHR, such as VA, number of intravitreal injections, and endophthalmitis, allowed for an evaluation of several of the clinical trial outcomes using the simulated prn treatment arms.
Despite the strengths of this novel methodology study, there are some limitations to consider.Because data from an EHR registry were used, clinical variables such as VA were not measured in a standardized fashion.In this study, VA was obtained within the IRIS Registry by extracting Snellen VA chart measurements sourced from the practice's EHR.Although these measurements are less standardized than those in a clinical trial, they are an accurate reflection of realworld clinical practice.Imaging data were not included and as a result, several anatomical inclusion and exclusion criteria in the CATT such as OCT and fluorescein angiography findings, as well as anatomical outcomes, could not be assessed. 8urthermore, while we used a structured approach to define prn and monthly treatment, we did not review clinical notes to confirm treatment regimen.Thus, it is possible that not all patients in the prn arm were intentionally being treated in a prn manner.Additionally, this study used age, gender, and baseline VA to perform exact matching and IPSW.There are several other relevant characteristics including insurance status, individual physician and practice preferences, and social determinants of health that can impact both agent choice and frequency of visits and treatment.19e21 A study that takes into account some of these factors could be a consideration for future work.
While there are limitations to acknowledge, the applications of this methodology are vast.The CATT was chosen for this study due to the large number of nAMD patients receiving anti-VEGF treatment for this condition, as well as the ability to access publicly-available deidentified patient-level clinical trial data.Both were necessary to apply these methodologies.In the future, there is an opportunity to apply these same methods to more recent clinical trials and across other disease states in ophthalmology, as well as the chance to integrate this methodology prospectively in future clinical trials.
The value of these methodologies could lie in the ability to expedite evaluation of therapeutic agents in a clinical trial setting for disease states that are less common or currently have no treatment if data used in clinical care were used to generate a synthetic control arm.For example, clinical trials in uncommon inherited retinal degenerations or neuroophthalmic conditions could potentially be streamlined significantly with the use of real-world control arms.Focusing on disease states that meet the qualifications of having no commercially available treatment could also further minimize the limitation noted in this study that nonrandom factors impact treatment choice and regimen in the real-world.The results of this paper could help stimulate discussion for future novel applications of real-world data to generate synthetic control arms in ophthalmology clinical trials.

Table 1 .
Cohort Attrition* for IRIS Registry prn Bevacizumab and Ranibizumab Cohorts CATT ¼ Comparison of AMD Treatment Trial; CNV ¼ choroidal neovascularization; ICD ¼ International Classification of Diseases; IRIS ¼ Intelligent Research in Sight; nAMD ¼ neovascular age-related macular degeneration; PDT ¼ photodynamic therapy; prn ¼ pro re nata.*Each number under patients and patient eyes represents the cumulative attrition after all criteria in that specific row and all preceding rows have been applied.Figure 1.Schematic diagram depicting the study design which compares 2 monthly treatment arms from the CATT trial to bevacizumab and ranibizumab prn treatment arms generated using IRIS Registry data and applying both exact matching and inverse propensity score weighting.Primary and secondary outcomes evaluated for each of these treatment arms are described to the right.CATT ¼ Comparison of AMD Treatment Trial; IPSW ¼ inverse propensity score weighting; IRIS ¼ Intelligent Research in Sight; nAMD ¼ neovascular age-related macular degeneration; prn ¼ pro re nata.

Table 3 .
Baseline Characteristics for prn Cohorts Created by Exact Matching and Inverse Propensity Score Weighting were significantly worse in the prn treatment arms from the IRIS Registry compared with the monthly treatment arms from the CATT for both bevacizumab and ranibizumab.The VA changes for the CATT treatment arms and the VA changes for the IRIS Registry cohort for both exact matching and IPSW are presented in Table4.All P-values comparing CATT monthly VA outcomes with IRIS prn VA outcomes for IPSW and exact matching were 0.001 for both ranibizumab and bevacizumab.AE 14.0 vs. IPSW: 2.8 letters AE 15.0 letters, P ¼ 0.43).Similarly, patient eyes in the bevacizumab prn treatment arm generated using exact matching gained 2.4 letters and patient eyes in the bevacizumab prn treatment arm generated using IPSW gained 2.1 letters (exact matching: 2.4 letters AE 15.4 vs. IPSW: 2.1 letters AE 16.0 letters, P ¼ 0.79).

Table 4 .
Clinical Outcomes for CATT Patients in prn and Monthly Cohorts and IRIS Registry Simulated prn Arms 4