Data from randomized controlled trials (RCTs) are the primary source for health technology assessment (HTA) however these are limited by strict patient inclusion criteria, leading to concerns about whether treatment benefit estimates are accurate for all patients (generalizability). Real-World Data (RWD) have been proposed as a solution however as these are observational data there is additional potential for bias when estimating treatment effectiveness. To maximize the utility of RWD it is useful to consider the whole process of evidence generation and robustly address issues of feasibility and validity.

A series of complementary studies investigated whether population-based routinely collected health data from Scotland are suitable for estimating the effectiveness of chemotherapy for early breast cancer. Firstly, a prognostic score was validated in this population. Secondly, a comparison of RWD and randomized trial effectiveness estimates was made to investigate feasibility and validity of several methods – Propensity Score Matching (PSM), Instrumental variables (IV) and Regression Discontinuity. Finally, effectiveness estimates in trial underrepresented groups were produced.

PSM and IV were feasible and produced results in relatively close agreement with randomized data. Effectiveness estimates in trial underrepresented groups (women over 70 years and women with high comorbidity) were consistent with an approximate one-third reduction in the risk of death from breast cancer. This is equivalent to approximately a 3–4 percentage point difference in all cause mortality over 10 years in these groups.

RWD are a feasible for generating estimates of effectiveness of adjuvant chemotherapy in early stage breast cancer. The process of using RWD for this purpose should include careful assessment of data quality and comparison of alternative strategies for causal identification in the context of available randomized data.