Causal Effect Estimation

Estimate the causal effect of an exposure, intervention, treatment, or policy on an outcome.

R1: Research task identification

  • Estimating causal effects for point exposures in populations

    Estimating the causal effect of an exposure, treatment, intervention, or policy (including its level, dose, or intensity) on an outcome in a specified population.

  • Estimating causal effects for sustained, dynamic, or complex exposure regimes in populations

    Estimating the causal effect of ongoing, dynamic, or sustained exposure, treatment, policy, or combination regimes (including their dose, duration, or intensity) on an outcome in a specified population.

  • Estimating heterogeneous causal effects
    Estimating how the causal effect of an exposure on an outcome varies across levels of one or more predefined variables

  • Causal mediation analyses
    Decomposing the causal effect of an exposure on an outcome into different pathways and/or estimating the impact of intervening on intermediate variables

I2: Identify estimand(s)

Carefully describe the target quantity/quantities of interest and all relevant criteria using the appropriate estimand framework

  1. Clinical decision, action, or policy to be informed by the effect estimate (e.g., treatment or policy guideline development, regulatory approval, health technology assessment, post-marketing safety evaluation)

  2. Target population (including the population definition and eligibility criteria)

  3. Start of follow-up (the specific event or decision point that marks the start of follow-up, e.g. date of diagnosis)

  4. Treatment conditions (including definitions of all treatments and comparator, and whether treatment is defined as a point intervention or a sustained or dynamic strategy)

  5. Endpoint (outcome definitions and end of follow-up)

  6. Summary measure (e.g. risk ratio, risk difference)

  7. Handling of intercurrent events* (definitions of all intercurrent events and descriptions of how will they be handled, e.g. deaths handled by treatment policy)

For heterogeneous causal effects and causal mediation analyses, see supplementary estimand specifications

G3: Gauge data fitness

Carefully describe the target quantity/quantities of interest and all relevant criteria using the appropriate estimand framework

Sample Requirements

  • The sample should be drawn from the target population, or contain sufficient information to enable transportation or reweighting of estimates to that population.

  • Sample size should be sufficient to estimate the estimand with desired precision, with adequate observations per confounder to avoid sparse data bias

  • Period of observation is sufficient to observe the outcome following exposure (e.g. follow-up long enough for the outcome to accrue, and sufficient prior observation time to establish baseline exposure status and exclude prevalent users)

Variable requirements

  • Exposure, outcome, any mediators, and all key confounders are available and accurately measured, with consistent definitions and coding practices across data sources, sites, and time periods.

  • Variables are measured with sufficient timing and frequency to establish the correct causal ordering between exposure and outcome

  • Exposure (and any mediators) varies within the sample and across all relevant confounding strata

O4: Outline and consider key sources of error, bias & threats to validity

Consider all potential sources of error, bias and threats to validity and outline mitigation strategies. Table 2 contains prompt questions to help identify major sources of bias and select mitigation strategies.

Selection

Type 2 selection bias (generalizability bias)
The sample is not a census or random sample from the target population, the distribution of certain effect modifiers differs between the sample and the target population, and the analytic sample cannot be reweighted to represent the target population (e.g., due to absence of survey weights or insufficient auxiliary data to construct them). Causal effect estimates will not generalise to the target population.

Type 1 selection bias (collider restriction bias)
When both the exposure and outcome of interest are related to presence in the data, whether directly or through shared or intermediate causes, the estimated causal effects will be biased. Common instances include Berkson's bias*, index event bias**, survivorship bias***, and M-bias****.

Measurement

Outcome measurement error
The outcome is measured with error, which may be unrelated to the exposure (non-differential) or related to the exposure (differential, e.g., due to surveillance bias, where diagnostic examination is more likely for a particular exposure). Non-differential error can lead to diluted effect estimates for categorical outcomes; differential error can introduce bias in any direction.

Exposure measurement error
The exposure is measured with error, which may be unrelated to the outcome (non-differential) or related to the outcome (differential). Non-differential error generally leads to diluted effect estimates; differential error can introduce bias in any direction.

Dependent measurement errors
There is correlated measurement error in both the exposure and outcome, e.g. because both variables are measured by the same clinician during the same clinical encounter. Dependent errors can introduce bias in any direction.

Effect modifier measurement error
The effect measure modifier of interest is measured with error, which can introduce bias in any direction in the apparent heterogeneity between groups.

Mediator measurement error
The mediator is measured with error, which can introduce bias in any direction in the apparent direct and/or indirect effects. 

Missing Data

Data are missing for the exposure, outcome, mediator, effect measure modifier, or other adjustment variables, and the probability of missingness is related to the exposure and/or outcome (e.g., due to an informative observation processes). When the analysis requires follow-up over time, loss to follow-up or informative censoring occurs when individuals leave the observation window for reasons related to the treatment or outcome (e.g., transferring care, death captured in a different system). Informative missingness can bias causal effect estimates in any direction.

Data Source Heterogeneity

Data are pooled from multiple healthcare systems or across calendar periods with different measurement practices and/or case-mix. This introduces uncertainty and can bias causal effect estimates when site or calendar period covaries with the exposure, outcome, or confounders.

Data Sparsity

Insufficient sample size, or strong determination of the exposure*, leads to poor overlap between exposure groups after conditioning, producing extreme weights, unstable coefficients, and biased effect estimates.

Footnote: *Poor covariate overlap may arise because the sample is too small to adequately represent all covariate patterns or because certain covariate patterns strongly predict exposure/intervention status. This second issue cannot be resolved by simply collecting more data.

Confounding

Unobserved baseline confounding
One or more baseline common causes of the exposure and outcome are not captured in the data, leading to biased effect estimates. Important instances in EHR include confounding by indication (where the clinical reason for prescribing a treatment itself influences the outcome status) and protopathic bias (where early undiagnosed symptoms of the outcome influence treatment status).

Residual baseline confounding
One or more available baseline confounders are poorly measured, or have been coarsened (e.g. dichotomised), meaning conditioning does not fully remove confounding (e.g., dichotomised ‘obesity’ does not capture confounding by BMI). Effect estimates will remain biased even after conditioning, though typically less so.

Unobserved time-varying confounding
For mediation analyses or studies of sustained or dynamic treatment regimes, one or more time-varying common causes of subsequent exposure and outcome are not captured in the data, leading to biased effect estimates.

Residual time-varying confounding
For mediation analyses or studies of sustained or dynamic treatment regimes, one or more available time-varying confounders are poorly measured or have been coarsened, meaning effect estimates will remain biased, even after appropriate handling, though typically less so.

Time Zero Alignment

Lead-time bias
When the exposure influences the timing of the index event (e.g., a screening intervention leads to earlier diagnosis), comparing time-to-event from the index event between exposed and unexposed groups may show an apparent benefit even if the exposure has no true effect on the outcome.

Immortal time bias
Exposure definitions that require a period of time to be satisfied (e.g., "at least 7 days of treatment") guarantee that exposed individuals have survived event-free for that period, while no equivalent guarantee exists for unexposed individuals. This creates a biased comparison by excluding early events from the exposed group.

Prevalent user bias
Studying an exposure that began before the observation window (prevalent use) means that any adverse outcomes occurring between exposure initiation and entry into the data have not been captured, selecting for individuals who survived and tolerated the exposure.

_________________

* Berkson’s bias = A type of selection bias that occurs when both primary variables of interest (e.g. an exposure and outcome) both directly influence entry into the sample

** Index event bias = A type of selection bias that occurs when a primary variable of interest (e.g. the outcome) is only possible among people who have experienced a qualifying event that is directly influenced by another variable of interest (e.g. the exposure), and the primary variable is also related to the qualifying event through shared causes. 

*** Survivorship bias = a type of selection bias that occurs when a primary variable of interest (e.g. the exposure) directly influences survival to study entry, and another variable of interest (e.g. the outcome) is also related to survival through shared causes.

**** M-bias = a type of selection bias that occurs when there are unmeasured causes of two primary variables of interest (e.g. the exposure and outcome) that also cause study entry. In EHR data, this often arises through informed presence bias, where presence in the dataset is influenced by factors (e.g. healthcare utilisation, socioeconomic position) that are also linked to the primary variables of interest.

R5: Run appropriate analysis

Select and conduct analyses that are suitable for estimating your target estimands in the available data

  • Choose identification strategy (e.g., backdoor/frontdoor adjustment, IV estimation, or other quasi-experimental approaches) and state the identification assumptions for the target estimand

  • Choose appropriate estimation approach for the estimand and identification strategy (e.g., for backdoor approach with point treatment: outcome regression, propensity score methods, or doubly-robust estimators; for IV approach: two-stage regression or ratio of coefficients; for sustained or time-varying treatments, or causal mediation analyses: g-computation or inverse probability weighting of marginal structural models*)

  • Identify variables that need conditioning, weighting, and/or standardizing using causal reasoning (e.g., a directed acyclic graph)

  • Choose and implement appropriate analytical methods for the competing event and intercurrent event strategies specified in the estimand (e.g., cause-specific or subdistribution models for competing events; g-methods for hypothetical strategies) 

  • For each source of error, bias, or validity threat identified in the O4 step (above), specify the analytical strategy and mitigation approach, documenting the details in Table 2 

  • Pre-specify falsification strategies, including negative control exposures and/or outcomes, to detect residual confounding or other biases

*Standard regression adjustment is inappropriate when time-varying confounders are affected by prior treatment, as conditioning on these variables simultaneously blocks causal pathways and introduces collider bias.

O6: Outline and assess assumptions

Clearly outline the assumptions behind your results and conduct appropriate sensitivity analyses

  • Exchangeability 

    Identify and highlight all possible ways that exchangeability may be violated. For point treatments, covariate balance diagnostics should be presented where possible. For sustained or dynamic treatment regimes, assess sequential exchangeability at each decision point where feasible. For mediation analyses, assess exchangeability of both the exposure-outcome and mediator-outcome relationships. Quantitative bias analyses should be performed where specific unmeasured confounders are identified but unavailable in the data.

  • Positivity

    Examine covariate overlap between treatment groups. For point treatments, assess the distribution of propensity scores or analytic weights for extreme values or high variability. For sustained or dynamic treatment regimes, assess weight distributions for evidence of near-positivity violations at each decision point where feasible. For mediation analyses, assess whether joint positivity holds for exposure-mediator combinations. Structural violations may indicate that certain causal effects are not reliably estimable from the available data.

  • Consistency

    Discuss whether there are multiple versions of the exposure that could differ in their effect on the outcome (e.g., different formulations, doses, or routes of administration captured under a single exposure definition), and how this might affect interpretation.

  • No interference

    State whether the assumption that one individual's treatment status does not affect another individual's outcome is plausible in the study context. Where interference is possible (e.g., infectious disease, shared healthcare resources), discuss the potential implications.

  • Strategy-specific assumptions

    Where identification relies on IV estimation or other quasi-experimental approaches, the additional assumptions of that strategy (e.g.  relevance and exclusion restriction for IV, parallel trends for difference-in-differences, etc.) should be explicitly stated, justified, and empirically evaluated (where possible)

  • Competing events and censoring assumptions:

    For time-to-event outcomes, state the assumptions underlying the chosen approach to competing events and censoring (e.g., non-informative censoring, independence of competing events), and discuss whether these assumptions are plausible given the causal structure.

  • Missing data assumptions

    State the assumed missing data mechanism (MCAR, MAR, or MNAR) and justify the chosen analytical approach. Assess sensitivity of estimates to missing data assumptions and analytical approach if missing data are substantial.

  • Model assumptions

    Report and check all model assumptions (e.g. linearity, proportional hazards) as appropriate

U7: Use appropriate language

Describe and interpret aims, methods, and results in terms of target estimands and estimated causal effects, avoiding associational, predictive, risk factor, or definitive causal language.

Examples:

  • Aim: ‘We aimed to estimate the causal effect of X* vs X on Y’

  • Methods: ‘The average treatment effect of X* vs X on Y was estimated using Method Z’

  • Results: ‘The estimated causal effect of X* vs X on Y was …’

  • Discussion: ‘Our findings suggest that exposure to X* vs X may have a small harmful causal effect on Y’

Examples to avoid:

  • Aim: ‘We aimed to estimate the association between X and Y adjusting for covariates Z’ (unclear language in the aim obscures the true aim and target estimand)

  • Methods: ‘We estimated the association between X and Y using Methods Z...’ (unclear language in the methods obscures the true aim and target estimand)

  • Results: ‘The association between X and Y was...’ (unclear language in the results obscures the true aim and target estimand)

  • Discussion: ‘Our findings suggest that X may have a modest association with Y’ (unclear language in the discussion obscures the true aim and target estimand) 

  • Throughout:

    • "X predicted higher levels of Y" (predictive language obscures causal aim and target estimand)

    • ‘X was a risk factor for Y’ (risk factor language is unclear and should be avoided)

    • ‘X caused Y’ (definitive causal language presents the estimate as an established fact rather than a quantitative estimate based on assumptions)

S8: Satisfy reporting and transparency standards

Follow current best practice and relevant reporting guidelines for reporting study details and results.

  • Pre-register a study protocol and statistical analysis plan (e.g. on OSF) before data access, clearly stating the causal effect estimations of interest. Consider prospective registration on a formal registry (e.g. EU PAS Register).

  • Make analytical code available (e.g., as a supplement to the publication, alongside the protocol, or in a public repository), having reviewed it for disclosive content

  • Provide a data availability statement describing the process for obtaining access to the source data. Report summary-level information including sample flow diagrams and baseline  sample characteristics

  • Follow RECORD reporting guidelines (or RECORD-PE for pharmacoepidemiology studies), along with appropriate design-specific or context-specific reporting and methodological guidance (TARGET guidelines for target trial emulations, AGREMA for mediation analyses, STROBE-MR for Mendelian randomisation studies, Tennant et al 2021 (IJE) for studies using DAGs to estimate causal effects through backdoor adjustment, Jandoc et al 2015 (J Clin Epi) for studies conducting interrupted time series analysis).