Advanced Statistics and Machine Learning for Health Research

Mark Khurana; Neil Scheidwasser

19 Reporting Standards, TRIPOD+AI, and Clinical Impact

19.1 Introduction

Thousands of clinical prediction models are published every year. Yet only a small fraction are ever validated externally, and fewer still make it into clinical practice. One of the major reasons is poor reporting. Many papers omit critical methodological details, overstate performance, or fail to describe the model in enough detail for anyone to reproduce or implement it. In 2024, the TRIPOD+AI statement was published to address this problem head-on. A reporting guideline is simply an agreed checklist of what a study must describe, so that readers can judge whether the results are trustworthy and whether the model could be used elsewhere.

This chapter walks you through the current reporting standards for clinical prediction models, explains why they matter, and gives you the tools to produce work that meets the expectations of top medical journals.

The lifecycle below frames the whole chapter. A prediction model is not finished when the paper is published. It must be reported clearly, implemented in real workflows, and monitored over time.

flowchart LR
    A[Development] --> B[Validation]
    B --> C[Transparent reporting<br/>TRIPOD+AI]
    C --> D[Implementation<br/>at the bedside]
    D --> E[Monitoring<br/>and updating]
    E -.-> A

Figure 19.1: The lifecycle of a clinical prediction model, from development to ongoing monitoring. Conceptual schematic aligned with the TRIPOD (Moons et al. 2015) and TRIPOD+AI (Collins et al. 2024) reporting frameworks.

19.2 Why reporting standards matter

Consider trying to use a published prediction model in your own clinical setting. You would need to know:

What predictors are included, and exactly how they were measured
What outcome was predicted, and at what time horizon
How missing data were handled
How the model was validated, and on what population
The model’s calibration, not just its discrimination
Whether the model would be feasible to use in your context

Remarkably, many published prediction model studies omit several of these details. That makes the model impossible to trust or to reuse. A systematic review by Collins et al. found that reporting quality was poor across most published prediction model studies, which motivated the development of formal reporting guidelines.

Why should a clinician care? Because a poorly reported model is one you cannot safely apply to your own patients. If you do not know how risk was defined, in whom it was measured, or whether the predicted percentages are honest, you cannot tell whether acting on the model will help or harm someone in front of you.

19.3 The TRIPOD+AI statement

The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guideline was first published in 2015. In 2024, it was updated as TRIPOD+AI to address machine learning and AI-based models.

In plain terms, TRIPOD+AI is a checklist of things every prediction-model paper should report. It lets a reader judge whether the model is trustworthy and could actually be used. Think of it the same way you think of CONSORT for randomised trials or STROBE for observational studies. These are not hurdles, but shared expectations that make your work credible and reproducible. Journals increasingly require it, and reviewers will look for it.

A related tool you may encounter is PROBAST (Prediction model Risk Of Bias ASsessment Tool). Where TRIPOD+AI tells authors what to report, PROBAST helps a reader formally assess whether a published model is at risk of bias, for example because of how patients were selected or how the model was validated. Reporting guidelines and risk-of-bias tools work hand in hand.

TRIPOD+AI applies to studies that develop, validate, or update a prediction model, whether based on regression, machine learning, or deep learning. The key additions in the +AI version include:

Fairness evaluation: checking whether the model performs equally well across demographic groups (for example, whether it predicts as accurately for women as for men, or across ethnic groups)
Open science: sharing code, data (where possible), and the model itself, so others can check and reuse it
Handling of AI-specific issues: hyperparameter tuning (the settings that control how a machine learning model learns), computational requirements, and software dependencies

19.3.1 Key TRIPOD+AI items

The full checklist contains items across title, abstract, introduction, methods, results, and discussion. Here are some of the most commonly missed items:

Study design and participants:

Describe the study design (e.g., cohort, case-control, registry)
Specify the eligibility criteria clearly
Report dates of recruitment and follow-up

Predictors and outcome:

List all candidate predictors, how they were defined, and when they were measured
Define the outcome precisely, including the time horizon for prognostic models

Missing data:

Report the amount of missing data for each variable
Describe the method used to handle missing data (complete case analysis, imputation)
If multiple imputation was used, report the number of imputations and imputation model

Model development:

Describe the modelling approach (logistic regression, random forest, etc.)
Report how continuous predictors were handled (linear terms, splines, categorised)
For ML models: report hyperparameter tuning strategy, software, and random seeds

Model performance:

Report discrimination (e.g., C-statistic) with confidence intervals
Report calibration (calibration plot, calibration slope and intercept)
Report clinical utility (decision curve analysis or net benefit) where relevant

Validation:

Describe the validation approach (internal, external, temporal)
Report performance in the validation data, not just the development data

If some of these terms are still fuzzy, here is the quick clinician’s translation. Discrimination (the C-statistic) is whether the model ranks a patient who has the outcome above one who does not — like a diagnostic test’s ability to separate cases from non-cases. Calibration is whether the predicted risks are numerically honest: when the model says “20% risk”, do roughly 20 in 100 such patients actually have the event? A model can discriminate well yet be badly calibrated (consistently over- or under-stating risk), which is dangerous if clinicians act on the actual numbers. Clinical utility (decision curve analysis / net benefit) goes one step further and asks whether using the model to guide decisions does more good than harm at a realistic risk threshold. A complete paper reports all three, in the validation data, not just the development data where performance is always flattering.

Exercise 1: TRIPOD+AI audit

Find a recently published clinical prediction model paper (try searching PubMed for “prediction model” in your area of interest). Using the TRIPOD+AI checklist, evaluate how many items the paper reports adequately.

Focus on:

Is the outcome clearly defined with a time horizon?
Is calibration reported, or only discrimination?
How was missing data handled?
Were continuous predictors appropriately modelled (splines) or just categorised?
Is the model available for others to use (coefficients, code, web calculator)?

Write a brief (one paragraph) critical appraisal.

Solution

Code

# =============================================================================
# Chapter 19 - Exercise 1: TRIPOD+AI audit of a published paper
# Completed for Capstone 1: Cardiovascular Risk Prediction Model
# =============================================================================

# This exercise asks you to complete the TRIPOD+AI checklist for one of the
# capstone projects. Below is a completed checklist for Capstone 1:
# "Development and external validation of a cardiovascular risk prediction
# model using Framingham methodology, validated in NHANES data."

checklist <- data.frame(
  Item = c(
    "1. Title",
    "2. Abstract",
    "3a. Background/rationale",
    "3b. Objectives",
    "4a. Source of data",
    "4b. Dates of study",
    "5a. Key eligibility criteria",
    "5b. Treatments received",
    "6a. Outcome definition",
    "6b. Outcome timing",
    "6c. Blinding of outcome",
    "7. Predictors",
    "8. Sample size",
    "9. Missing data",
    "10a. Statistical analysis: model development",
    "10b. Model specification",
    "10c. Predictor selection",
    "10d. Model performance measures",
    "11. Risk groups",
    "12a. Internal validation",
    "12b. External validation",
    "13. Fairness assessment (AI-specific)",
    "14. Results: participants",
    "15. Results: model development",
    "16. Results: model performance",
    "17. Results: model updating",
    "18. Discussion: interpretation",
    "19. Discussion: limitations",
    "20. Discussion: implications",
    "21. Supplementary: code and data",
    "AI-1. Data preprocessing",
    "AI-2. Hyperparameter tuning",
    "AI-3. Model explainability",
    "AI-4. Software and hardware"
  ),
  Section = c(
    "Title page",
    "Abstract",
    "Introduction",
    "Introduction",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Methods",
    "Results",
    "Results",
    "Results",
    "Results",
    "Discussion",
    "Discussion",
    "Discussion",
    "Supplement",
    "Methods/Supplement",
    "Methods/Supplement",
    "Results/Supplement",
    "Methods"
  ),
  How_Addressed = c(
    # 1. Title
    "'Development and external validation of a logistic regression model
    for predicting 10-year cardiovascular disease risk: a Framingham-NHANES
    study.' Identifies study type (development + external validation) and
    modelling approach.",

    # 2. Abstract
    "Structured abstract with: objective, study design, data sources,
    outcome (10-year CVD), predictors, sample sizes, C-statistic and
    calibration results for both development and validation cohorts.",

    # 3a. Background
    "Existing CVD risk models (Framingham, QRISK, SCORE2) and their
    limitations. Gap: need for updated validation in contemporary US
    population.",

    # 3b. Objectives
    "To develop a 10-year CVD risk prediction model using Framingham
    data and externally validate it in NHANES 2017-2020.",

    # 4a. Source of data
    "Development: Framingham Heart Study teaching dataset (riskCommunicator
    R package). Validation: NHANES 2017-2020 (nhanesA package). Both are
    publicly available.",

    # 4b. Dates
    "Framingham: original cohort with follow-up through 2005. NHANES:
    2017-2020 survey cycles.",

    # 5a. Eligibility
    "Adults aged 30-74, free of CVD at baseline, with complete data on
    key predictors. Exclude prior MI, stroke, heart failure.",

    # 5b. Treatments
    "Not applicable (prediction model, not treatment comparison).
    However, blood pressure treatment status is included as a predictor.",

    # 6a. Outcome definition
    "10-year cardiovascular event: composite of MI, coronary death,
    stroke, or heart failure requiring hospitalisation.",

    # 6b. Outcome timing
    "10-year follow-up from baseline examination. Censored at death
    from non-CVD causes or loss to follow-up.",

    # 6c. Blinding
    "Not applicable for retrospective analysis. Outcome ascertainment
    in Framingham was by adjudication committee blinded to risk factors.",

    # 7. Predictors
    "Pre-specified based on established Framingham model: age, sex, total
    cholesterol, HDL cholesterol, systolic blood pressure, blood pressure
    treatment status, smoking, diabetes. No data-driven selection.",

    # 8. Sample size
    "Sample size justified using Riley et al. (2020) criteria via
    pmsampsize package: 8 predictors, anticipated outcome prevalence 10%,
    target R-squared 0.15, requiring minimum ~800 events.",

    # 9. Missing data
    "Development cohort: <5% missing. Validation: up to 12% for some
    laboratory values. Handled by multiple imputation (MICE, 50 datasets,
    20 iterations). Complete-case sensitivity analysis performed.",

    # 10a. Model development
    "Logistic regression for 10-year risk. Pre-specified predictors,
    no variable selection. Continuous predictors modelled with restricted
    cubic splines (4 knots for age and SBP; 3 knots for cholesterol)
    based on prior literature.",

    # 10b. Model specification
    "Full model equation provided in supplementary materials. Coefficients
    table with 95% CIs in main text.",

    # 10c. Predictor selection
    "All predictors pre-specified from clinical knowledge. No stepwise
    selection. Rationale provided for each predictor.",

    # 10d. Performance measures
    "Discrimination: C-statistic (Harrell's concordance) with 95% CI.
    Calibration: calibration plot (observed vs predicted), calibration
    slope, calibration-in-the-large. Both reported for development
    (bootstrap-corrected) and external validation.",

    # 11. Risk groups
    "Patients categorised into low (<5%), moderate (5-10%), high (10-20%),
    and very high (>20%) 10-year CVD risk groups. Classification tables
    provided.",

    # 12a. Internal validation
    "500 bootstrap resamples for optimism-corrected C-statistic and
    calibration slope. Apparent and corrected performance reported.",

    # 12b. External validation
    "Model applied to NHANES data without recalibration. C-statistic
    and calibration plot in external data. Recalibration of intercept
    also explored.",

    # 13. Fairness
    "C-statistic and calibration reported separately by sex (male/female)
    and by race/ethnicity (non-Hispanic White, non-Hispanic Black,
    Hispanic, Asian). Differences in performance >0.05 C-statistic
    flagged for discussion.",

    # 14. Participants
    "Flow diagram showing sample selection: patients screened, excluded
    (with reasons), and included in development and validation cohorts.
    Table 1 with baseline characteristics stratified by CVD status.",

    # 15. Model development
    "Full regression table with coefficients, standard errors, odds
    ratios, 95% CIs, and p-values. Presented as Table 2.",

    # 16. Model performance
    "Table 3: discrimination (C-statistic) and calibration metrics for
    development (apparent and corrected) and external validation.
    Figures: calibration plots (development and validation), decision
    curve analysis.",

    # 17. Model updating
    "Recalibration of intercept in NHANES improved calibration-in-the-
    large from X to Y. Full model updating was explored but not
    recommended due to limited improvement.",

    # 18. Interpretation
    "Model performance compared with published Framingham, QRISK3, and
    PCE models. Clinical utility assessed via decision curve analysis
    across clinically relevant threshold probabilities (5-20%).",

    # 19. Limitations
    "Limitations discussed: (1) Framingham cohort may not represent
    contemporary diverse populations; (2) NHANES validation limited by
    shorter follow-up and self-reported outcomes; (3) Missing data
    assumptions; (4) Ecological fallacy in subgroup performance
    assessment.",

    # 20. Implications
    "Clinical implications for CVD risk assessment. Recommendation for
    recalibration before use in non-US populations. Need for prospective
    validation.",

    # 21. Code and data
    "R code provided via GitHub repository [URL]. Framingham data
    available via riskCommunicator package. NHANES data publicly available.
    Reproducible via renv for package management.",

    # AI-1. Data preprocessing
    "Not applicable (logistic regression, not ML/AI). However, data
    cleaning steps, outlier handling, and transformation of predictors
    (restricted cubic splines) described in supplement.",

    # AI-2. Hyperparameter tuning
    "Not applicable for logistic regression. Number of knots for RCS
    pre-specified based on Harrell's recommendations.",

    # AI-3. Model explainability
    "Not applicable (logistic regression is inherently interpretable).
    Coefficient interpretation provided. Nomogram included for clinical
    use.",

    # AI-4. Software
    "R version 4.4.1. Packages: rms (v6.8), survival (v3.7), mice
    (v3.16), gtsummary (v1.7), pmsampsize (v1.1), riskCommunicator
    (v1.0), nhanesA (v1.1). Full renv.lock file in repository."
  ),

  Applicable = c(
    "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes",
    "Partially (no treatment comparison)",
    "Yes", "Yes",
    "Partially (retrospective)",
    "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes",
    "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes",
    "No (not ML)", "No (not ML)", "No (not ML)", "Yes"
  ),

  stringsAsFactors = FALSE
)

# Print the checklist
cat(strrep("=", 70), "\n")
cat("TRIPOD+AI Checklist - Capstone 1: CV Risk Prediction Model\n")
cat(strrep("=", 70), "\n\n")

for (i in 1:nrow(checklist)) {
  cat("ITEM:", checklist$Item[i], "\n")
  cat("  Section:", checklist$Section[i], "\n")
  cat("  Applicable:", checklist$Applicable[i], "\n")
  cat("  How addressed:", trimws(checklist$How_Addressed[i]), "\n\n")
}

# Summary
cat("\n=== SUMMARY ===\n")
n_applicable <- sum(checklist$Applicable == "Yes")
n_partial <- sum(grepl("Partially", checklist$Applicable))
n_na <- sum(checklist$Applicable == "No (not ML)")

cat("Items fully addressed:", n_applicable, "\n")
cat("Items partially applicable:", n_partial, "\n")
cat("Items not applicable (ML-specific):", n_na, "\n")
cat("Total items:", nrow(checklist), "\n")

cat("\nNote: AI-specific items (AI-1 through AI-3) are not applicable because\n")
cat("this capstone uses logistic regression, not machine learning. If a\n")
cat("gradient boosted model or neural network were used instead, these\n")
cat("items would need to be addressed with details on feature engineering,\n")
cat("hyperparameter search strategy, and SHAP/LIME explanations.\n")

Code

# =============================================================================
# Chapter 19 - Exercise 1: TRIPOD+AI audit of a published paper
# Completed for Capstone 1: Cardiovascular Risk Prediction Model
# =============================================================================

# This exercise asks you to complete the TRIPOD+AI checklist for one of the
# capstone projects. Below is a completed checklist for Capstone 1:
# "Development and external validation of a cardiovascular risk prediction
# model using Framingham methodology, validated in NHANES data."

checklist = [
    {
        "item": "1. Title",
        "section": "Title page",
        "applicable": "Yes",
        "how": (
            "'Development and external validation of a logistic regression "
            "model for predicting 10-year cardiovascular disease risk: a "
            "Framingham-NHANES study.' Identifies study type (development + "
            "external validation) and modelling approach."
        )
    },
    {
        "item": "2. Abstract",
        "section": "Abstract",
        "applicable": "Yes",
        "how": (
            "Structured abstract with: objective, study design, data sources, "
            "outcome (10-year CVD), predictors, sample sizes, C-statistic and "
            "calibration results for both development and validation cohorts."
        )
    },
    {
        "item": "3a. Background/rationale",
        "section": "Introduction",
        "applicable": "Yes",
        "how": (
            "Existing CVD risk models (Framingham, QRISK, SCORE2) and their "
            "limitations. Gap: need for updated validation in contemporary "
            "US population."
        )
    },
    {
        "item": "3b. Objectives",
        "section": "Introduction",
        "applicable": "Yes",
        "how": (
            "To develop a 10-year CVD risk prediction model using Framingham "
            "data and externally validate it in NHANES 2017-2020."
        )
    },
    {
        "item": "4a. Source of data",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Development: Framingham Heart Study teaching dataset "
            "(riskCommunicator R package). Validation: NHANES 2017-2020 "
            "(nhanesA package). Both publicly available."
        )
    },
    {
        "item": "4b. Dates of study",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Framingham: original cohort with follow-up through 2005. "
            "NHANES: 2017-2020 survey cycles."
        )
    },
    {
        "item": "5a. Key eligibility criteria",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Adults aged 30-74, free of CVD at baseline, with complete data "
            "on key predictors. Exclude prior MI, stroke, heart failure."
        )
    },
    {
        "item": "5b. Treatments received",
        "section": "Methods",
        "applicable": "Partially",
        "how": (
            "Not applicable (prediction model, not treatment comparison). "
            "However, BP treatment status included as a predictor."
        )
    },
    {
        "item": "6a. Outcome definition",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "10-year CVD event: composite of MI, coronary death, stroke, "
            "or heart failure requiring hospitalisation."
        )
    },
    {
        "item": "6b. Outcome timing",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "10-year follow-up from baseline examination. Censored at "
            "death from non-CVD causes or loss to follow-up."
        )
    },
    {
        "item": "6c. Blinding of outcome",
        "section": "Methods",
        "applicable": "Partially",
        "how": (
            "Retrospective analysis. Framingham outcome adjudication was "
            "by committee blinded to risk factors."
        )
    },
    {
        "item": "7. Predictors",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Pre-specified: age, sex, total cholesterol, HDL cholesterol, "
            "systolic BP, BP treatment, smoking, diabetes. No data-driven "
            "selection."
        )
    },
    {
        "item": "8. Sample size",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Justified using Riley et al. (2020) criteria via pmsampsize: "
            "8 predictors, prevalence 10%, target R^2 0.15, requiring "
            "minimum ~800 events."
        )
    },
    {
        "item": "9. Missing data",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Development: <5% missing. Validation: up to 12%. Handled by "
            "MICE (50 datasets, 20 iterations). Complete-case sensitivity "
            "analysis performed."
        )
    },
    {
        "item": "10a. Model development",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Logistic regression for 10-year risk. Pre-specified predictors. "
            "Continuous predictors with restricted cubic splines."
        )
    },
    {
        "item": "10b. Model specification",
        "section": "Methods",
        "applicable": "Yes",
        "how": "Full model equation in supplementary materials."
    },
    {
        "item": "10c. Predictor selection",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "All predictors pre-specified from clinical knowledge. "
            "No stepwise selection."
        )
    },
    {
        "item": "10d. Performance measures",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Discrimination: C-statistic with 95% CI. Calibration: "
            "calibration plot, calibration slope, calibration-in-the-large."
        )
    },
    {
        "item": "11. Risk groups",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Low (<5%), moderate (5-10%), high (10-20%), very high (>20%) "
            "10-year CVD risk categories."
        )
    },
    {
        "item": "12a. Internal validation",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "500 bootstrap resamples for optimism-corrected C-statistic "
            "and calibration slope."
        )
    },
    {
        "item": "12b. External validation",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Model applied to NHANES without recalibration. C-statistic "
            "and calibration reported. Recalibration of intercept explored."
        )
    },
    {
        "item": "13. Fairness assessment (AI-specific)",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Performance reported by sex and race/ethnicity. Differences "
            ">0.05 C-statistic flagged."
        )
    },
    {
        "item": "14. Results: participants",
        "section": "Results",
        "applicable": "Yes",
        "how": (
            "Flow diagram with sample selection. Table 1 stratified by "
            "CVD event status."
        )
    },
    {
        "item": "15. Results: model development",
        "section": "Results",
        "applicable": "Yes",
        "how": (
            "Full regression table with coefficients, ORs, 95% CIs, "
            "and p-values."
        )
    },
    {
        "item": "16. Results: model performance",
        "section": "Results",
        "applicable": "Yes",
        "how": (
            "Table of discrimination and calibration metrics. Calibration "
            "plots and decision curve analysis figures."
        )
    },
    {
        "item": "17. Results: model updating",
        "section": "Results",
        "applicable": "Yes",
        "how": (
            "Recalibration of intercept in NHANES. Full model updating "
            "explored but not recommended."
        )
    },
    {
        "item": "18. Discussion: interpretation",
        "section": "Discussion",
        "applicable": "Yes",
        "how": (
            "Comparison with Framingham, QRISK3, PCE models. Clinical "
            "utility via decision curve analysis."
        )
    },
    {
        "item": "19. Discussion: limitations",
        "section": "Discussion",
        "applicable": "Yes",
        "how": (
            "Framingham may not represent diverse populations. NHANES "
            "validation limited by follow-up. Missing data assumptions."
        )
    },
    {
        "item": "20. Discussion: implications",
        "section": "Discussion",
        "applicable": "Yes",
        "how": (
            "Clinical implications for CVD risk assessment. Need for "
            "recalibration in non-US populations."
        )
    },
    {
        "item": "21. Code and data",
        "section": "Supplement",
        "applicable": "Yes",
        "how": (
            "Code on GitHub. Data publicly available via R packages. "
            "Reproducible via renv/conda."
        )
    },
    {
        "item": "AI-1. Data preprocessing",
        "section": "Methods/Supplement",
        "applicable": "No (logistic regression)",
        "how": (
            "Not applicable for logistic regression. Data cleaning and "
            "spline transformations described in supplement."
        )
    },
    {
        "item": "AI-2. Hyperparameter tuning",
        "section": "Methods/Supplement",
        "applicable": "No (logistic regression)",
        "how": (
            "Not applicable. Number of RCS knots pre-specified based on "
            "Harrell's recommendations."
        )
    },
    {
        "item": "AI-3. Model explainability",
        "section": "Results/Supplement",
        "applicable": "No (logistic regression)",
        "how": (
            "Not applicable (logistic regression is inherently "
            "interpretable). Nomogram provided."
        )
    },
    {
        "item": "AI-4. Software and hardware",
        "section": "Methods",
        "applicable": "Yes",
        "how": (
            "Python 3.11 with scikit-learn, lifelines, statsmodels, "
            "tableone. Full requirements.txt in repository."
        )
    },
]

# --- Print the checklist ---
print("=" * 70)
print("TRIPOD+AI Checklist")
print("Capstone 1: Cardiovascular Risk Prediction Model")
print("=" * 70)

for entry in checklist:
    print(f"\nITEM: {entry['item']}")
    print(f"  Section: {entry['section']}")
    print(f"  Applicable: {entry['applicable']}")
    print(f"  How addressed: {entry['how']}")

# --- Summary ---
n_yes = sum(1 for e in checklist if e['applicable'] == 'Yes')
n_partial = sum(1 for e in checklist if 'Partially' in e['applicable'])
n_no = sum(1 for e in checklist if e['applicable'].startswith('No'))

print(f"\n{'=' * 70}")
print("SUMMARY")
print(f"{'=' * 70}")
print(f"Items fully addressed:               {n_yes}")
print(f"Items partially applicable:          {n_partial}")
print(f"Items not applicable (ML-specific):  {n_no}")
print(f"Total items:                         {len(checklist)}")

print("""
Note on AI-specific items:
AI-specific items (AI-1 through AI-3) are not applicable because this
capstone uses logistic regression, not machine learning. If a gradient
boosted model or neural network were used, these items would require:
  - AI-1: Feature engineering pipeline, normalization, encoding
  - AI-2: Hyperparameter search strategy (grid, random, Bayesian)
  - AI-3: SHAP values, partial dependence plots, or LIME explanations

The TRIPOD+AI checklist is available from the EQUATOR Network:
https://www.equator-network.org/reporting-guidelines/tripod-ai/

Key points for completing TRIPOD+AI:
1. NEVER report AUROC/C-statistic without calibration
2. Always include a calibration plot
3. Fairness assessment across demographic subgroups is now required
4. Code and data availability statements are mandatory
5. For AI/ML models, explainability methods must be described
""")

19.4 From model to bedside: the implementation gap

Implementation (also called deployment) means actually putting a model to work in routine care, for example building it into the electronic health record so it flags high-risk patients automatically. This is the step where most models fail. Even well-developed, well-validated models face an uphill battle to reach the bedside. Smits, van Kuijk & Wynants (2026) dedicate five chapters of their book to this topic, covering the journey from model selection through innovation development, impact evaluation, and implementation.

This matters to clinicians because a model that never reaches the bedside, or that reaches it in an unusable form, helps no patient. The gap between “the model works on paper” and “the model improves care” is where good ideas are usually lost.

19.4.1 Key barriers to implementation

The model doesn’t address a real clinical need. Models built for academic interest rather than to solve a genuine decision problem rarely get implemented.
The model isn’t embedded in clinical workflow. A model that requires a clinician to manually enter 15 variables into a spreadsheet will not be used, regardless of how accurate it is. Equally, a model that fires too many warnings causes alert fatigue: clinicians become so used to seeing pop-up alerts that they start ignoring them, including the important ones.
Performance hasn’t been demonstrated in the target population. External validation in the specific setting where the model will be used is essential.
Clinicians don’t trust the model. Transparency, explainability, and evidence of clinical utility (not just statistical performance) build trust.
There is no plan for maintenance. Models can degrade over time as populations and practices change. This slow loss of accuracy is often called calibration drift: the model’s predicted risks gradually stop matching the real event rates (for example, a sepsis model that becomes too pessimistic after treatment protocols change). A plan for ongoing monitoring and updating is essential.

19.4.2 The prediction model-based innovation (PMBI) framework

Smits et al. (2026) introduce the concept of a PMBI. The idea is that the prediction model itself is only the engine. What clinicians actually use is the complete tool wrapped around it, including:

The user interface (how predictions are displayed)
Decision support (what actions are recommended at different risk levels)
Integration with electronic health records
Training materials for end users
A monitoring and updating plan

A growing convention is to publish a model card: a short, standardised summary sheet describing what the model predicts, who it was built for, how it performed, and its known limitations. Think of it as a “package insert” for a prediction model, written so that a busy clinician can quickly judge whether it is fit for their setting.

The final step is an impact study: research that tests whether using the model in practice actually changes outcomes for patients (for example, fewer missed diagnoses or shorter hospital stays), rather than only measuring statistical accuracy on paper.

Note

Key insight: Building a good prediction model is necessary but not sufficient for clinical impact. The model must be embedded in a workable clinical tool, validated in the target setting, and demonstrated to improve patient outcomes.

19.5 Risk communication

A prediction is only useful if the person receiving it understands it. Presenting predicted probabilities to patients and clinicians is not straightforward, and how a number is phrased can change the decision it leads to. Research in health literacy shows that:

Frequencies are easier to understand than probabilities. “3 out of 100 patients like you” is clearer than “3% probability.”
Visual aids help. Icon arrays (showing 100 faces with 3 highlighted) are effective.
Framing matters. “97% chance of survival” feels different from “3% chance of death” — both are true.
Uncertainty should be communicated. Presenting a point estimate without a confidence interval overstates precision.

19.5.1 Presenting results to different audiences

Audience	Recommended format
Patients	Frequencies, icon arrays, plain language
Clinicians	Risk categories with action thresholds, decision curves
Researchers	Full performance metrics, calibration plots, code
Policymakers	Population-level impact, cost-effectiveness

19.6 Ethical considerations

19.6.1 Algorithmic bias

Prediction models can perpetuate or amplify existing health disparities if:

Training data underrepresents certain populations
Predictors serve as proxies for protected characteristics (e.g., postcode as proxy for race/ethnicity)
Performance is not evaluated across subgroups

TRIPOD+AI specifically requires fairness evaluation: reporting model performance stratified by relevant demographic groups.

19.6.3 The EU AI Act and regulatory considerations

The European Union’s AI Act (2024) classifies medical AI applications as “high risk” and imposes requirements for transparency, human oversight, and documentation. Clinical prediction models that qualify as medical devices may need regulatory approval (e.g., CE marking in the EU, FDA clearance in the US).

19.7 Writing a statistical methods section

A well-written methods section for a prediction model study should include:

Study design and setting (one paragraph)
Participants — eligibility criteria, dates, sample size (one paragraph)
Predictors — list, definitions, measurement timing (one paragraph)
Outcome — definition, ascertainment, time horizon (one paragraph)
Missing data — amount, mechanism assumed, handling method (one paragraph)
Model development — method, how continuous variables were handled, variable selection approach (one paragraph)
Model performance — discrimination, calibration, clinical utility measures (one paragraph)
Validation — internal and/or external validation approach (one paragraph)
Software — R/Python version, key packages, random seed (one sentence)

Exercise 2: Write a methods section

Using the prediction model you developed in earlier chapters (or a hypothetical one), write a complete statistical methods section following the structure above. Aim for approximately 500 words.

Check your methods section against the TRIPOD+AI checklist. Are all key items covered?

Solution

Code

# =============================================================================
# Chapter 19 - Exercise 2: Write a Statistical Methods Section
# Retrospective cohort study: SGLT2i vs DPP4i and MACE in T2DM
# =============================================================================

# This is a conceptual exercise. The statistical methods section is provided
# as a multi-line character string that could be included in a manuscript.

methods_section <- '
STATISTICAL METHODS

Study Design and Population
This was a retrospective cohort study of 5,000 adults with type 2 diabetes
mellitus identified from the hospital electronic health records (EHR) database
between 1 January 2015 and 31 December 2023. Patients were eligible if they
had a new prescription for either an SGLT2 inhibitor or a DPP-4 inhibitor, with
no prior use of the comparator drug class. Patients with a history of major
adverse cardiovascular events (MACE) prior to the index date were excluded.
The index date (time zero) was defined as the date of first prescription of
the study drug, consistent with a new-user, active comparator design to
minimise immortal time bias and confounding by indication.

Primary and Secondary Outcomes
The primary outcome was time to first MACE, defined as a composite of
myocardial infarction (ICD-10: I21), ischaemic stroke (ICD-10: I63), or
cardiovascular death (underlying cause of death codes I00-I99). Patients were
followed from the index date until the first MACE event, death from non-
cardiovascular causes, loss to follow-up, end of the study period
(31 December 2023), or 5 years after the index date, whichever occurred first.

Sample Size
With 5,000 patients and an anticipated event rate of 8% over 5 years in the
DPP-4 inhibitor group, the study had approximately 80% power to detect a
hazard ratio of 0.70 or smaller at a two-sided alpha of 0.05, assuming a
1:1 treatment group ratio and accounting for 10% loss to follow-up.

Descriptive Statistics
Baseline characteristics were summarised as means (SD) for normally
distributed continuous variables, medians (IQR) for skewed continuous
variables, and frequencies (percentages) for categorical variables.
Standardised mean differences (SMDs) were used to compare baseline
characteristics between treatment groups, with an absolute SMD < 0.1
indicating adequate balance; p-values were not used for baseline comparisons
in accordance with current recommendations.

Propensity Score Estimation and Matching
The propensity score -- the probability of receiving an SGLT2 inhibitor
versus a DPP-4 inhibitor -- was estimated using multivariable logistic
regression. Covariates included age, sex, body mass index (BMI), glycated
haemoglobin (HbA1c), estimated glomerular filtration rate (eGFR), history
of cardiovascular disease, hypertension, and smoking status. These covariates
were selected a priori based on clinical knowledge and a directed acyclic
graph (DAG) encoding assumed causal relationships. Propensity scores were
used for 1:1 nearest-neighbour matching without replacement, using a caliper
of 0.2 standard deviations of the logit of the propensity score. Covariate
balance after matching was assessed using SMDs, with all covariates required
to achieve an absolute SMD < 0.1. Propensity score overlap was assessed
visually using density plots.

Primary Analysis
The primary analysis estimated the average treatment effect on the treated
(ATT) using a Cox proportional hazards regression model fitted to the matched
cohort, with SGLT2 inhibitor use as the sole covariate. The proportional
hazards assumption was tested using scaled Schoenfeld residuals and
log-log survival plots. If the assumption was violated, a time-varying
coefficient or restricted mean survival time (RMST) analysis was planned
as an alternative. Hazard ratios (HRs) with 95% confidence intervals (CIs)
were reported.

Missing Data
Missing covariate data ranged from 2% (age) to 15% (BMI). Missingness was
assumed to be missing at random (MAR) conditional on observed variables.
Multiple imputation by chained equations (MICE) was performed with 50
imputed datasets and 20 iterations per dataset. The imputation model
included all analysis variables (covariates, exposure, outcome indicator,
and the Nelson-Aalen cumulative hazard estimate) to ensure compatibility
with the substantive analysis model. Propensity score estimation and
matching were performed within each imputed dataset, and results were
pooled using Rubin rules. A complete-case analysis was performed as a
sensitivity analysis.

Sensitivity Analyses
Five pre-specified sensitivity analyses were conducted: (1) complete-case
analysis; (2) inverse probability of treatment weighting (IPTW) with
stabilised weights as an alternative to matching; (3) inclusion of
additional covariates (income quintile, number of medications) in the
propensity score model; (4) restriction to patients with at least
12 months of follow-up; and (5) E-value calculation to quantify the
minimum strength of association an unmeasured confounder would need with
both the treatment and the outcome to explain away the observed association.

Multiple Comparisons
As this study had a single pre-specified primary outcome, no adjustment
for multiple comparisons was applied to the primary analysis. Secondary
outcomes were interpreted with appropriate caution as hypothesis-generating.

Software
All analyses were conducted using R version 4.4.1 (R Foundation for
Statistical Computing, Vienna, Austria) with the following packages:
MatchIt (v4.5.5) for propensity score matching, cobalt (v4.5.1) for
balance assessment, survival (v3.7-0) for Cox regression, mice (v3.16.0)
for multiple imputation, and survey (v4.4-2) for IPTW analyses. The
random seed was set to 42 for reproducibility. Analysis code is available
at [repository URL]. Two-sided p-values < 0.05 were considered
statistically significant.
'

# Print the methods section
cat(methods_section)

# --- Checklist verification ---
cat("\n\n=== CHECKLIST: Key Items Addressed ===\n")
cat("1. Study design and population:           YES\n")
cat("2. Primary and secondary outcomes:        YES\n")
cat("3. Sample size / power calculation:        YES\n")
cat("4. Descriptive statistics approach:        YES\n")
cat("5. Primary analysis model:                YES\n")
cat("6. Assumptions and how checked:           YES (PH assumption)\n")
cat("7. Missing data (extent & handling):       YES (MICE, 50 datasets)\n")
cat("8. Sensitivity analyses:                  YES (5 pre-specified)\n")
cat("9. Multiple comparisons:                  YES (single primary)\n")
cat("10. Software and versions:                YES\n")
cat("11. Active comparator, new-user design:   YES\n")
cat("12. DAG for covariate selection:          YES\n")
cat("13. Balance assessment (SMDs):            YES\n")
cat("14. E-value for unmeasured confounding:   YES\n")

Code

# =============================================================================
# Chapter 19 - Exercise 2: Write a Statistical Methods Section
# Retrospective cohort study: SGLT2i vs DPP4i and MACE in T2DM
# =============================================================================

# This is a conceptual exercise. The statistical methods section is provided
# as a detailed text output.

methods_section = """
STATISTICAL METHODS

Study Design and Population
This was a retrospective cohort study of 5,000 adults with type 2 diabetes
mellitus identified from the hospital electronic health records (EHR) database
between 1 January 2015 and 31 December 2023. Patients were eligible if they
had a new prescription for either an SGLT2 inhibitor or a DPP-4 inhibitor,
with no prior use of the comparator drug class. Patients with a history of
major adverse cardiovascular events (MACE) prior to the index date were
excluded. The index date (time zero) was defined as the date of first
prescription of the study drug, consistent with a new-user, active comparator
design to minimise immortal time bias and confounding by indication.

Primary and Secondary Outcomes
The primary outcome was time to first MACE, defined as a composite of
myocardial infarction (ICD-10: I21), ischaemic stroke (ICD-10: I63), or
cardiovascular death (underlying cause of death codes I00-I99). Patients were
followed from the index date until the first MACE event, death from non-
cardiovascular causes, loss to follow-up, end of the study period
(31 December 2023), or 5 years after the index date, whichever occurred first.

Sample Size
With 5,000 patients and an anticipated event rate of 8% over 5 years in the
DPP-4 inhibitor group, the study had approximately 80% power to detect a
hazard ratio of 0.70 or smaller at a two-sided alpha of 0.05, assuming a
1:1 treatment group ratio and accounting for 10% loss to follow-up.

Descriptive Statistics
Baseline characteristics were summarised as means (SD) for normally
distributed continuous variables, medians (IQR) for skewed continuous
variables, and frequencies (percentages) for categorical variables.
Standardised mean differences (SMDs) were used to compare baseline
characteristics between treatment groups, with an absolute SMD < 0.1
indicating adequate balance; p-values were not used for baseline comparisons
in accordance with current recommendations.

Propensity Score Estimation and Matching
The propensity score -- the probability of receiving an SGLT2 inhibitor
versus a DPP-4 inhibitor -- was estimated using multivariable logistic
regression. Covariates included age, sex, body mass index (BMI), glycated
haemoglobin (HbA1c), estimated glomerular filtration rate (eGFR), history
of cardiovascular disease, hypertension, and smoking status. These covariates
were selected a priori based on clinical knowledge and a directed acyclic
graph (DAG) encoding assumed causal relationships. Propensity scores were
used for 1:1 nearest-neighbour matching without replacement, using a caliper
of 0.2 standard deviations of the logit of the propensity score. Covariate
balance after matching was assessed using SMDs, with all covariates required
to achieve an absolute SMD < 0.1. Propensity score overlap was assessed
visually using density plots.

Primary Analysis
The primary analysis estimated the average treatment effect on the treated
(ATT) using a Cox proportional hazards regression model fitted to the matched
cohort, with SGLT2 inhibitor use as the sole covariate. The proportional
hazards assumption was tested using scaled Schoenfeld residuals and
log-log survival plots. If the assumption was violated, a time-varying
coefficient or restricted mean survival time (RMST) analysis was planned
as an alternative. Hazard ratios (HRs) with 95% confidence intervals (CIs)
were reported.

Missing Data
Missing covariate data ranged from 2% (age) to 15% (BMI). Missingness was
assumed to be missing at random (MAR) conditional on observed variables.
Multiple imputation by chained equations (MICE) was performed with 50
imputed datasets and 20 iterations per dataset. The imputation model
included all analysis variables (covariates, exposure, outcome indicator,
and the Nelson-Aalen cumulative hazard estimate) to ensure compatibility
with the substantive analysis model. Propensity score estimation and
matching were performed within each imputed dataset, and results were
pooled using Rubin's rules. A complete-case analysis was performed as a
sensitivity analysis.

Sensitivity Analyses
Five pre-specified sensitivity analyses were conducted: (1) complete-case
analysis; (2) inverse probability of treatment weighting (IPTW) with
stabilised weights as an alternative to matching; (3) inclusion of
additional covariates (income quintile, number of medications) in the
propensity score model; (4) restriction to patients with at least
12 months of follow-up; and (5) E-value calculation to quantify the
minimum strength of association an unmeasured confounder would need with
both the treatment and the outcome to explain away the observed association.

Multiple Comparisons
As this study had a single pre-specified primary outcome, no adjustment
for multiple comparisons was applied to the primary analysis. Secondary
outcomes were interpreted with appropriate caution as hypothesis-generating.

Software
All analyses were conducted using Python version 3.11 with the following
packages: scikit-learn (v1.4.0) for propensity score estimation, lifelines
(v0.29.0) for Cox proportional hazards regression, statsmodels (v0.14.1)
for weighted regression analyses, and tableone (v0.9.1) for baseline
characteristics tables. The random seed was set to 42 for reproducibility.
Analysis code is available at [repository URL]. Two-sided p-values < 0.05
were considered statistically significant.
"""

print(methods_section)

# --- Checklist verification ---
print("=" * 60)
print("CHECKLIST: Key Items Addressed in Methods Section")
print("=" * 60)

checklist = [
    ("Study design and population", True),
    ("Primary and secondary outcomes (defined)", True),
    ("Sample size / power calculation", True),
    ("Descriptive statistics approach", True),
    ("Primary analysis model (Cox PH)", True),
    ("Assumptions and how checked (PH assumption)", True),
    ("Missing data (extent, mechanism, handling)", True),
    ("Sensitivity analyses (5 pre-specified)", True),
    ("Multiple comparisons addressed", True),
    ("Software and package versions", True),
    ("New-user, active comparator design", True),
    ("DAG for covariate selection", True),
    ("Balance assessment method (SMDs)", True),
    ("E-value for unmeasured confounding", True),
    ("Caliper specification for matching", True),
    ("Number of imputations and iterations", True),
]

for item, addressed in checklist:
    status = "YES" if addressed else "NO"
    print(f"  [{status:>3}] {item}")

print("\nWord count: approximately 550 words")
print("(The exercise requests 250-400 words; this is comprehensive")
print("to cover all required items. Can be condensed for journals")
print("with strict word limits by moving details to a supplement.)")

19.8 The future of clinical prediction

Several trends are shaping the next decade of clinical prediction modelling:

Dynamic prediction models that update as new patient data become available (e.g., during a hospital stay)
Federated learning that trains models across institutions without sharing patient data
Foundation models adapted for clinical tasks (large language models, multimodal models)
Continuous model monitoring with automated detection of performance degradation
Patient-facing prediction tools integrated into health apps and patient portals

Regardless of the technology, the fundamentals covered in this course — proper validation, calibration assessment, clinical utility evaluation, and transparent reporting — will remain essential.

Exercise 3: Critical appraisal

Find a prediction model paper from 2024 or 2025 in a journal relevant to your field. Answer the following:

What clinical question does the model address?
Was the model developed with regression, ML, or both?
Was calibration reported? If so, was it assessed using a calibration plot?
Was clinical utility (decision curve analysis) reported?
Could you implement this model in your own setting with the information provided?
Does the paper comply with TRIPOD+AI?

Discuss your findings with a colleague or in the course discussion forum.

19.9 References and Further Reading

For prediction model reporting and development, see Smits et al. (2026), Van Calster et al. (2025), and Steyerberg (2019).
For decision curve analysis, see Vickers et al. (2019).
For regulatory context, see European Parliament and Council of the European Union (2024) (official text available at eur-lex.europa.eu).
For communicating risk to patients, see Gigerenzer and Edwards (2003).

Collins, Gary S, Karel G M Moons, Paula Dhiman, et al. 2024. “TRIPOD+AI Statement: Updated Guidance for Reporting Clinical Prediction Models That Use Regression or Machine Learning Methods.” BMJ 385: e078378. https://doi.org/10.1136/bmj-2023-078378.

European Parliament and Council of the European Union. 2024. EU Artificial Intelligence Act (Regulation 2024/1689). Official text available at eur-lex.europa.eu.

Gigerenzer, Gerd, and Adrian Edwards. 2003. “Simple Tools for Understanding Risks: From Innumeracy to Insight.” BMJ 327 (7417): 741–44. https://doi.org/10.1136/bmj.327.7417.741. Classic paper on risk communication.

Moons, Karel G M, Douglas G Altman, Johannes B Reitsma, et al. 2015. “Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement.” Annals of Internal Medicine 162 (1): 55–63.

Smits, Luc J M, Sander M J van Kuijk, and Laure Wynants. 2026. Improving Health Care with Clinical Prediction Models: From Idea to Impact. Maastricht University Press. https://doi.org/10.26481/mup.2603. CC BY 4.0

Steyerberg, Ewout W. 2019. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2nd ed. Springer.

Van Calster, Ben, Gary S Collins, Andrew J Vickers, et al. 2025. “Evaluation of Performance Measures in Predictive Artificial Intelligence Models to Support Medical Decisions: Overview and Guidance.” The Lancet Digital Health 7: e100916. https://doi.org/10.1016/j.landig.2025.100916.

Vickers, Andrew J, Ben van Calster, and Ewout W Steyerberg. 2019. “A Simple, Step-by-Step Guide to Interpreting Decision Curve Analysis.” Diagnostic and Prognostic Research 3: 18. https://doi.org/10.1186/s41512-019-0064-7.

19.1 Introduction

19.2 Why reporting standards matter

19.3 The TRIPOD+AI statement

19.3.1 Key TRIPOD+AI items

19.4 From model to bedside: the implementation gap

19.4.1 Key barriers to implementation

19.4.2 The prediction model-based innovation (PMBI) framework

19.5 Risk communication

19.5.1 Presenting results to different audiences

19.6 Ethical considerations

19.6.1 Algorithmic bias

19.6.2 Informed consent and transparency

19.6.3 The EU AI Act and regulatory considerations

19.7 Writing a statistical methods section

19.8 The future of clinical prediction

19.9 References and Further Reading