O6.
Outline & Assess Assumptions
Every EHR analysis depends on assumptions. These assumptions may concern representativeness, positivity, exchangeability, measurement, missing data, censoring, competing events, model specification, temporal stability, or deployment validity.
This step asks researchers to state those assumptions clearly, assess whether they are plausible, and conduct sensitivity analyses where appropriate.
Description
-
Assess whether the sample is representative of the target population for the health state or event of interest. Discuss potential sources of discrepancy between sample and target population. Consider whether standardisation or weighting to a reference population is appropriate.
-
Confirm that all strata of interest are sufficiently observed in the data. Discuss implications of sparse or empty strata for model-based descriptive estimates (e.g. standardization, MAIHDA, small area estimation).
-
For time-to-event outcomes, state the assumptions underlying the chosen approach to competing events and censoring (e.g., non-informative censoring, independence of competing events).
-
State the assumed missing data mechanism (MCAR, MAR, or MNAR) and justify the chosen analytical approach. Assess sensitivity of estimates to missing data assumptions and analytical approach if missing data are substantial.
-
Report and check all model-specific assumptions where model-based approaches are used (e.g., linearity, distributional assumptions in multilevel models).
Signal Discovery
-
State and justify the assumption that the chosen confounding adjustment strategy (e.g., principal component adjustment for population stratification, covariate adjustment for confounding by indication) is sufficient. Where residual confounding is plausible, assess the potential impact on findings using quantitative bias analysis.
-
State the assumed dependence structure between tests and justify the chosen error rate control procedure (FDR vs FWER). Assess sensitivity of findings to the choice of correction approach where feasible.
-
Where replication in independent data has been conducted, assess whether signals replicate and discuss potential reasons for non-replication. Where replication has not been conducted, acknowledge the preliminary nature of the findings and discuss the plausibility of the signals in light of existing evidence.
-
For time-to-event outcomes, state the assumptions underlying the chosen approach to competing events and censoring (e.g., non-informative censoring, independence of competing events).
-
State the assumed missing data mechanism (MCAR, MAR, or MNAR) and justify the chosen analytical approach. Assess sensitivity of findings to missing data assumptions and analytical approach if missing data are substantial.
-
Report and check all model-specific assumptions (e.g. linearity, distributional assumptions).
Factual Prediction
-
Assess whether the case-mix, predictor distributions, and outcome prevalence in the development data are representative of the intended deployment population. Consider whether learned predictor-outcome associations may depend on selection mechanisms or measurement processes specific to the development setting. Discuss potential sources of miscalibration when applying the model to new settings. Evaluate model performance in external validation samples where available.
-
Assess whether predictor-outcome relationships are likely to remain stable over time, considering changes in coding practices and evolving patient populations.
-
Confirm that all model predictors are realistically available at the intended point of prediction in clinical practice and that no predictors require a period of observation to be measured. Discuss implications if predictors are missing or measured differently in deployment settings.
-
For time-to-event outcomes, state the assumptions underlying the chosen approach to competing events and censoring (e.g., non-informative censoring, independence of competing events) and discuss whether these assumptions are plausible in the intended deployment context.
-
State the assumed missing data mechanism (MCAR, MAR, or MNAR) and justify the chosen analytical approach. Assess sensitivity of estimates to missing data assumptions and analytical approach if missing data are substantial.
-
Report and check all model-specific assumptions (e.g. linearity, proportional hazards for survival models).
Counterfactual Prediction
-
Identify and highlight all possible ways that exchangeability may be violated. For single point interventions, covariate balance diagnostics should be presented where possible. For sustained or dynamic intervention regimes, assess sequential exchangeability at each decision point where feasible. Quantitative bias analyses should be performed where specific unmeasured confounders are identified but unavailable in the data.
-
Examine covariate overlap between intervention groups. For single point interventions, assess the distribution of propensity scores or analytic weights for extreme values or high variability. For sustained or dynamic intervention regimes, or when multiple interventions are compared, assess weight distributions for evidence of near-positivity violations at each decision point where feasible. Structural violations may indicate that certain counterfactual predictions are not reliably estimable from the available data.
-
Discuss whether there are multiple versions of the intervention that could differ in their effect on the outcome (e.g., different formulations, doses, or routes of administration captured under a single intervention definition), and how this might affect the counterfactual predictions.
-
State whether the assumption that one individual's intervention does not affect another individual's outcome is plausible in the study context. Where interference is likely, discuss the implications for the counterfactual predictions.
-
Assess whether the causal effects underpinning the counterfactual predictions are likely to hold in the intended deployment population. Consider whether effect modifiers are distributed differently between the development and deployment populations, and whether selection mechanisms or measurement processes specific to the development setting may limit validity in the deployment context.
-
Assess whether the intervention-outcome relationships underpinning the counterfactual predictions are likely to remain stable over time, considering changes in clinical practice, coding practices, and evolving patient populations.
-
For time-to-event outcomes, state the assumptions underlying the chosen approach to competing events and censoring (e.g., non-informative censoring, independence of competing events), and discuss whether these assumptions are plausible given the causal structure.
-
State the assumed missing data mechanism (MCAR, MAR, or MNAR) and justify the chosen analytical approach. Assess sensitivity of predictions to missing data assumptions and analytical approach if missing data are substantial.
-
Report and check all model-specific assumptions (e.g., correct specification of the outcome model and treatment model in g-formula or IPW implementations).
Causal Effect Estimation
-
Identify and highlight all possible ways that exchangeability may be violated. For point treatments, covariate balance diagnostics should be presented where possible. For sustained or dynamic treatment regimes, assess sequential exchangeability at each decision point where feasible. For mediation analyses, assess exchangeability of both the exposure-outcome and mediator-outcome relationships. Quantitative bias analyses should be performed where specific unmeasured confounders are identified but unavailable in the data.
-
Examine covariate overlap between treatment groups. For point treatments, assess the distribution of propensity scores or analytic weights for extreme values or high variability. For sustained or dynamic treatment regimes, assess weight distributions for evidence of near-positivity violations at each decision point where feasible. For mediation analyses, assess whether joint positivity holds for exposure-mediator combinations. Structural violations may indicate that certain causal effects are not reliably estimable from the available data.
-
Discuss whether there are multiple versions of the exposure that could differ in their effect on the outcome (e.g., different formulations, doses, or routes of administration captured under a single exposure definition), and how this might affect interpretation.
-
State whether the assumption that one individual's treatment status does not affect another individual's outcome is plausible in the study context. Where interference is possible (e.g., infectious disease, shared healthcare resources), discuss the potential implications.
-
Where identification relies on IV estimation or other quasi-experimental approaches, the additional assumptions of that strategy (e.g., relevance and exclusion restriction for IV, parallel trends for difference-in-differences, etc.) should be explicitly stated, justified, and empirically evaluated (where possible)
-
For time-to-event outcomes, state the assumptions underlying the chosen approach to competing events and censoring (e.g., non-informative censoring, independence of competing events), and discuss whether these assumptions are plausible given the causal structure.
-
State the assumed missing data mechanism (MCAR, MAR, or MNAR) and justify the chosen analytical approach. Assess sensitivity of estimates to missing data assumptions and analytical approach if missing data are substantial.
-
Report and check all model assumptions (e.g., linearity, proportional hazards) as appropriate
By the end of this step, you should have:
Listed the assumptions required for the analysis and interpretation
Explained why each assumption is plausible or uncertain
Identified assumptions that can be empirically assessed
Planned sensitivity analyses where assumptions may be violated
Clarified how assumption violations would affect interpretation