O4: Outline and consider key sources of error, bias & threats to validity

Consider all potential sources of error, bias and threats to validity and outline mitigation strategies. Table 2 contains prompt questions to help identify major sources of bias and select mitigation strategies.

Selection

Type 2 selection bias (generalisability bias)
The sample is not a census or random sample from the target population, the distribution of certain effect modifiers differs between the sample and the target population, and the analytic sample cannot be reweighted to represent the target population. Detected signals may not replicate in the broader population, or true signals may be missed due to insufficient representation of relevant subgroups.

Type 1 selection bias (collider restriction bias)
When both the scanning variable (exposure, genotype, or phenotype) and the outcome of interest are related to presence in the data, whether directly or through shared or intermediate causes, spurious signals may be detected. Common instances include Berkson's bias*, index event bias**, survivorship bias***, and M-bias****.

Measurement

Outcome measurement error
The outcome is measured with error, which may be non-differential or differential. Non-differential error can attenuate true associations below detection thresholds after multiple testing correction; differential error can generate spurious signals or mask true signals.

Scanning variable measurement error
The scanning variable (exposure, genotype, or phenotype) is measured with error, which may be non-differential or differential. Non-differential error attenuates associations, increasing the risk of missed signals; differential error can generate spurious signals in any direction.

Dependent measurement errors
There is correlated measurement error in both the scanning variable and outcome, e.g. because both are derived from the same clinical documentation process. Dependent errors can generate spurious signals or mask true signals.

Missing Data

Data are missing for the outcome, scanning variable, or covariates, and the probability of missingness is related to the signal of interest. When the analysis requires follow-up over time, loss to follow-up or informative censoring occurs when individuals leave the observation window for reasons related to the scanning variable or outcome. Informative missingness can generate spurious signals or mask true signals.

Data Source Heterogeneity

Data are pooled from multiple healthcare systems or across calendar periods with different measurement practices and/or case-mix. This introduces uncertainty and can generate spurious signals or mask true signals when these differences covary with the scanning variable or outcome.

Data Sparsity

Insufficient observations for specific exposure-phenotype combinations leads to unstable association estimates, inflated effect sizes (i.e. ‘winners curse’), and an excess of false positives.

Confounding

Unobserved baseline confounding
One or more common causes of the scanning variable and outcome are not captured in the data, generating spurious signals. Important instances include population stratification in GWAS (where ancestry-related genetic variation correlates with both the variant and the outcome through shared environmental or demographic pathways), confounding by indication in pharmacovigilance, and shared lifestyle or environmental confounders in ExWAS.

Residual baseline confounding
One or more available baseline confounders are poorly measured or have been coarsened (e.g. dichotomised), meaning conditioning does not fully remove confounding. For example, principal component adjustment for population stratification in GWAS may not fully capture ancestry structure, leading to residual confounding by population stratification.

Time Zero Alignment

Lead-time bias
When the scanning variable influences the timing of the index event (e.g., a drug triggers earlier diagnostic investigation), spurious signals may be detected that reflect differential detection timing rather than genuine associations with the outcome.

Immortal time bias
Scanning variable definitions that require a period of time to be satisfied (e.g., "at least 7 days of drug exposure") guarantee that exposed individuals have survived event-free for that period, potentially generating spurious protective signals or masking harmful signals.

Prevalent user bias
Scanning for signals among prevalent users of an exposure means that adverse outcomes occurring before the observation window are uncaptured, selecting for individuals who survived and tolerated the exposure. This can mask true signals or attenuate genuine associations.

___________________________

* Berkson’s bias = A type of selection bias that occurs when both primary variables of interest (e.g. an exposure and outcome) both directly influence entry into the sample

** Index event bias = A type of selection bias that occurs when a primary variable of interest (e.g. the outcome) is only possible among people who have experienced a qualifying event that is directly influenced by another variable of interest (e.g. the exposure), and the primary variable is also related to the qualifying event through shared causes. 

*** Survivorship bias = a type of selection bias that occurs when a primary variable of interest (e.g. the exposure) directly influences survival to study entry, and another variable of interest (e.g. the outcome) is also related to survival through shared causes.

**** M-bias = a type of selection bias that occurs when there are unmeasured causes of two primary variables of interest (e.g. the exposure and outcome) that also cause study entry. In EHR data, this often arises through informed presence bias, where presence in the dataset is influenced by factors (e.g. healthcare utilisation, socioeconomic position) that are also linked to the primary variables of interest.