O4: Outline and consider key sources of error, bias & threats to validity

Consider all potential sources of error, bias and threats to validity and outline mitigation strategies. Table 2 contains prompt questions to help identify major sources of bias and select mitigation strategies.

Selection

Type 2 selection bias (generalizability bias)
The sample is not a census or random sample from the target population, the distribution of certain effect modifiers differs between the sample and the target population, and the analytic sample cannot be reweighted to represent the target population (e.g., due to absence of survey weights or insufficient auxiliary data to construct them). Counterfactual predictions will not generalise to the intended deployment population.

Type 1 selection bias (collider restriction bias)
When both the hypothetical intervention and outcome of interest are related to presence in the data, whether directly or through shared or intermediate causes, the predicted outcomes under the intervention will be biased. Common instances include Berkson's bias*, index event bias**, survivorship bias***, and M-bias****.

Measurement

Outcome measurement error
The outcome is measured with systematic error, meaning the model learns to predict the outcome under the specified intervention conditions incorrectly.

Intervention measurement error
The intervention variables are measured with error, meaning the counterfactual predictions reflect a different intervention than the one specified and may be biased in any direction. 

Dependent measurement errors
There is correlated measurement error in both the intervention variables and outcome, e.g. because both are measured by the same clinician during the same clinical encounter. Dependent errors can bias counterfactual predictions in any direction.

Missing Data

Data are missing for the outcome, intervention variables, or other adjustment variables, and the probability of missingness is related to the outcome and/or intervention (e.g., due to informative observation processes). When the analysis requires follow-up over time, loss to follow-up or informative censoring occurs when individuals leave the observation window for reasons related to the intervention or outcome (e.g., transferring care, death captured in a different system). Informative missingness can bias counterfactual predictions in any direction.

Data Source Heterogeneity

Data are pooled from multiple healthcare systems or across calendar periods with different measurement practices and/or case-mix. The estimated causal effects underpinning the counterfactual predictions may reflect an average across heterogeneous settings, biasing predictions for any specific deployment context.

Data sparsity

Insufficient sample size, or strong determination of one or more interventions of interest*, leads to few observations for certain intervention-covariate combinations, producing unstable counterfactual predictions.

Footnote: *Poor covariate overlap may arise because the sample is too small to adequately represent all covariate patterns or because certain covariate patterns strongly predict exposure/intervention status. This second issue cannot be resolved by simply collecting more data.

Confounding

Unobserved baseline confounding
One or more baseline common causes of the intervention and outcome are not captured in the data, leading to biased predictions. Important instances in EHR include confounding by indication (where the clinical reason for prescribing an intervention itself influences the outcome status) and protopathic bias (where early undiagnosed symptoms of the outcome influence intervention status).

Residual baseline confounding
One or more available baseline confounders are poorly measured, or have been coarsened (e.g., dichotomised), meaning conditioning does not fully remove confounding (e.g., dichotomised ‘obesity’ does not capture confounding by BMI). Predictions will remain biased, even after conditioning, though typically less so.

Unobserved time-varying confounding
For predictions involving multiple interventions, or sustained or dynamic intervention regimes, one or more time-varying common causes of subsequent intervention decisions and outcome are not captured in the data, leading to biased predictions.

Residual time-varying confounding
For predictions involving multiple interventions, or sustained or dynamic intervention regimes, one or more available time-varying confounders are poorly measured or have been coarsened, meaning predictions will remain biased even after appropriate handling, though typically less so.

Time Zero Alignment

Lead-time bias
When the hypothetical intervention influences the timing of the index event (e.g., a screening intervention leads to earlier diagnosis), counterfactual predictions of time-to-event from the index event may overestimate the benefit of the intervention.

Immortal time bias
Intervention definitions that require a period of time to be satisfied (e.g., "at least 7 days of treatment") guarantee that individuals assigned to that intervention have survived event-free for that period, biasing the counterfactual predictions in favour of the intervention.

Prevalent user bias
Basing predictions on prevalent users of the intervention means that adverse outcomes occurring before the observation window are uncaptured. The counterfactual predictions reflect a selected population of survivors and tolerators rather than all individuals who would initiate the intervention.

_______________________

* Berkson’s bias = A type of selection bias that occurs when both primary variables of interest (e.g. an exposure and outcome) both directly influence entry into the sample

** Index event bias = A type of selection bias that occurs when a primary variable of interest (e.g. the outcome) is only possible among people who have experienced a qualifying event that is directly influenced by another variable of interest (e.g. the exposure), and the primary variable is also related to the qualifying event through shared causes. 

*** Survivorship bias = a type of selection bias that occurs when a primary variable of interest (e.g. the exposure) directly influences survival to study entry, and another variable of interest (e.g. the outcome) is also related to survival through shared causes.

**** M-bias = a type of selection bias that occurs when there are unmeasured causes of two primary variables of interest (e.g. the exposure and outcome) that also cause study entry. In EHR data, this often arises through informed presence bias, where presence in the dataset is influenced by factors (e.g. healthcare utilisation, socioeconomic position) that are also linked to the primary variables of interest.