Clinical prediction modeling in the era of AI: a blessing and a curse
MaartenvanSmeden
777 views
107 slides
Aug 22, 2024
Slide 1 of 107
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
About This Presentation
Keynote for the 8th Danish Bioinformatics conference in Kopenhagen, 22 August 2024.
Slide Content
Maarten van Smeden, PhD
Julius Center forHealth Sciences andPrimaryCare
8th Annual Danish Bioinformatics conference
Kopenhagen, 22 August 2024
Clinicalpredictionmodeling
in theera of AI:
a blessinganda curse
Julius Center forHealth Sciences andPrimaryCare
8th Annual Danish Bioinformatics conference
Kopenhagen, 22 August 2024
Clinicalpredictionmodeling
in theera of AI:
a blessinganda curse
Disclosures
•Nothing to disclose
•Nothing to disclose
In this lecture, I will talk about….
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
AI
BLESSINGS
AND
CURSES
AI
BLESSINGS
AND
CURSES
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Imgsource: https://www.topbots.com/generative-vs-predictive-ai/
Kopenhagen, 22 Aug 2024 @MaartenvSmedenDe Hond et al, Lancet Digital Health, 2024
Prediction
Source: https://www.intellspot.com/unsupervised-vs-supervised-learning/#google_vignette
Source: https://www.intellspot.com/unsupervised-vs-supervised-learning/#google_vignette
Kopenhagen, 22 Aug 2024 @MaartenvSmedenvan Smeden et al., JCE, 2021, doi: 10.1016/j.jclinepi.2021.01.009
Adversarialexample
https://bit.ly/2N4mQFo; https://bit.ly/2W7X9rF
https://bit.ly/2N4mQFo; https://bit.ly/2W7X9rF
Kopenhagen, 22 Aug 2024 @MaartenvSmedenhttps://tinyurl.com/3knkuzs3
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Image source: https://shorturl.at/styGJ
Image source: https://shorturl.at/styGJ
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
APGAR score
Apgar et al. JAMA, 1958
APGAR score
Apgar et al. JAMA, 1958
Virginia Apgar (1909- 1974)
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Still commonly used, but…
Sources: doi: 10.1097/ANC.0000000000000859, 10.1136/bmj.38117.665197.F7
Still commonly used, but…
Sources: doi: 10.1097/ANC.0000000000000859, 10.1136/bmj.38117.665197.F7
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
“65% of U.S. physiciansusedMDCalcon a weeklybasis”
“65% of U.S. physiciansusedMDCalcon a weeklybasis”
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Landscape of clinicalpredictionmodels
•42 models for kidney failure in chronic kidney disease (Ramspek, 2019)
•40 models for incident heart failure (Sahle, 2017)
•37 models for treatment response in pulmonary TB (Peetluk, 2021)
•35 models for in vitro fertilisation (Ratna, 2020)
•34 models for stroke in type-2 diabetes (Chowdhury, 2019)
•34 models for graft failure in kidney transplantation (Kabore, 2017)
•31 models for length of stay in ICU (Verburg, 2016)
•30 models for low back pain (Haskins, 2015)
•27 models for pediatric early warning systems (Trubey, 2019)
•27 models for malaria prognosis (Njim, 2019)
•26 models for postoperative outcomes colorectal cancer (Souwer, 2020)
•26 models for childhood asthma (Kothalawa, 2020)
•25 models for lung cancer risk (Gray, 2016)
•25 models for re-admissionafteradmittedforheartfailure (Mahajan, 2018)
•23 models for recovery after ischemic stroke (Jampathong, 2018)
•23 models for delirium in older adults (Lindroth, 2018)
•21 models for atrial fibrillation detection in community (Himmelreich, 2020)
•19 models for survival after resectable pancreatic cancer (Stijker, 2019)
•18 models for recurrence hep. carc. after liver transplant (Al-Ameri, 2020)
•18 models for future hypertension in children (Hamoen, 2018)
•18 models for risk of falls after stroke (Walsh, 2016)
•18 models for mortality in acute pancreatitis (Di, 2016)
•17 models for bacterial meningitis (van Zeggeren, 2019)
•17 models for cardiovascular disease in hypertensive population (Cai, 2020)
•14 models for ICU delirium risk (Chen, 2020)
•14 models for diabetic retinopathy progression (Haider, 2019)
•1382 models for cardiovascular disease (Wessler, 2021)
•731 models related to COVID-19 (Wynants, 2020)
•408 models for COPD prognosis (Bellou, 2019)
•363 models for cardiovascular disease general population (Damen, 2016)
•327 models for toxicity prediction after radiotherapy (Takada, 2022)
•263 prognosis models in obstetrics (Kleinrouweler, 2016)
•258 models mortality after general trauma (Munter, 2017)
•160 female-specific models for cardiovascular disease (Baart, 2019)
•142 models for mortality prediction in preterm infants (van Beek, 2021)
•119 models for critical care prognosis in LMIC (Haniffa, 2018)
•101 models for primary gastric cancer prognosis (Feng, 2019)
•99 models for neck pain (Wingbermühle, 2018)
•81 models for sudden cardiac arrest (Carrick, 2020)
•74 models for contrast-induced acute kidney injury (Allen, 2017)
•73 models for 28/30 day hospital readmission (Zhou, 2016)
•68 models for preeclampsia (De Kat, 2019)
•68 models for living donor kidney/iver transplant counselling (Haller, 2022)
•67 models for traumatic brain injury prognosis (Dijkland, 2019)
•64 models for suicide / suicide attempt (Belsher, 2019)
•61 models for dementia (Hou, 2019)
•58 modelsforbreastcancerprognosis(Phung, 2019)
•52 modelsforpre‐eclampsia(Townsend, 2019)
•52 modelsforcolorectalcancerrisk (Usher-Smith, 2016)
•48 modelsforincident hypertension(Sun, 2017)
•46 models for melanoma (Kaiser, 2020)
•46 models for prognosis after carotid revascularisation (Volkers, 2017)
•43 models for mortality in critically ill (Keuning, 2019)
Landscape of clinicalpredictionmodels
•42 models for kidney failure in chronic kidney disease (Ramspek, 2019)
•40 models for incident heart failure (Sahle, 2017)
•37 models for treatment response in pulmonary TB (Peetluk, 2021)
•35 models for in vitro fertilisation (Ratna, 2020)
•34 models for stroke in type-2 diabetes (Chowdhury, 2019)
•34 models for graft failure in kidney transplantation (Kabore, 2017)
•31 models for length of stay in ICU (Verburg, 2016)
•30 models for low back pain (Haskins, 2015)
•27 models for pediatric early warning systems (Trubey, 2019)
•27 models for malaria prognosis (Njim, 2019)
•26 models for postoperative outcomes colorectal cancer (Souwer, 2020)
•26 models for childhood asthma (Kothalawa, 2020)
•25 models for lung cancer risk (Gray, 2016)
•25 models for re-admissionafteradmittedforheartfailure (Mahajan, 2018)
•23 models for recovery after ischemic stroke (Jampathong, 2018)
•23 models for delirium in older adults (Lindroth, 2018)
•21 models for atrial fibrillation detection in community (Himmelreich, 2020)
•19 models for survival after resectable pancreatic cancer (Stijker, 2019)
•18 models for recurrence hep. carc. after liver transplant (Al-Ameri, 2020)
•18 models for future hypertension in children (Hamoen, 2018)
•18 models for risk of falls after stroke (Walsh, 2016)
•18 models for mortality in acute pancreatitis (Di, 2016)
•17 models for bacterial meningitis (van Zeggeren, 2019)
•17 models for cardiovascular disease in hypertensive population (Cai, 2020)
•14 models for ICU delirium risk (Chen, 2020)
•14 models for diabetic retinopathy progression (Haider, 2019)
•1382 models for cardiovascular disease (Wessler, 2021)
•731 models related to COVID-19 (Wynants, 2020)
•408 models for COPD prognosis (Bellou, 2019)
•363 models for cardiovascular disease general population (Damen, 2016)
•327 models for toxicity prediction after radiotherapy (Takada, 2022)
•263 prognosis models in obstetrics (Kleinrouweler, 2016)
•258 models mortality after general trauma (Munter, 2017)
•160 female-specific models for cardiovascular disease (Baart, 2019)
•142 models for mortality prediction in preterm infants (van Beek, 2021)
•119 models for critical care prognosis in LMIC (Haniffa, 2018)
•101 models for primary gastric cancer prognosis (Feng, 2019)
•99 models for neck pain (Wingbermühle, 2018)
•81 models for sudden cardiac arrest (Carrick, 2020)
•74 models for contrast-induced acute kidney injury (Allen, 2017)
•73 models for 28/30 day hospital readmission (Zhou, 2016)
•68 models for preeclampsia (De Kat, 2019)
•68 models for living donor kidney/iver transplant counselling (Haller, 2022)
•67 models for traumatic brain injury prognosis (Dijkland, 2019)
•64 models for suicide / suicide attempt (Belsher, 2019)
•61 models for dementia (Hou, 2019)
•58 modelsforbreastcancerprognosis(Phung, 2019)
•52 modelsforpre‐eclampsia(Townsend, 2019)
•52 modelsforcolorectalcancerrisk (Usher-Smith, 2016)
•48 modelsforincident hypertension(Sun, 2017)
•46 models for melanoma (Kaiser, 2020)
•46 models for prognosis after carotid revascularisation (Volkers, 2017)
•43 models for mortality in critically ill (Keuning, 2019)
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Landscape of clinicalpredictionmodels
•42 models for kidney failure in chronic kidney disease (Ramspek, 2019)
•40 models for incident heart failure (Sahle, 2017)
•37 models for treatment response in pulmonary TB (Peetluk, 2021)
•35 models for in vitro fertilisation (Ratna, 2020)
•34 models for stroke in type-2 diabetes (Chowdhury, 2019)
•34 models for graft failure in kidney transplantation (Kabore, 2017)
•31 models for length of stay in ICU (Verburg, 2016)
•30 models for low back pain (Haskins, 2015)
•27 models for pediatric early warning systems (Trubey, 2019)
•27 models for malaria prognosis (Njim, 2019)
•26 models for postoperative outcomes colorectal cancer (Souwer, 2020)
•26 models for childhood asthma (Kothalawa, 2020)
•25 models for lung cancer risk (Gray, 2016)
•25 models for re-admissionafteradmittedforheartfailure (Mahajan, 2018)
•23 models for recovery after ischemic stroke (Jampathong, 2018)
•23 models for delirium in older adults (Lindroth, 2018)
•21 models for atrial fibrillation detection in community (Himmelreich, 2020)
•19 models for survival after resectable pancreatic cancer (Stijker, 2019)
•18 models for recurrence hep. carc. after liver transplant (Al-Ameri, 2020)
•18 models for future hypertension in children (Hamoen, 2018)
•18 models for risk of falls after stroke (Walsh, 2016)
•18 models for mortality in acute pancreatitis (Di, 2016)
•17 models for bacterial meningitis (van Zeggeren, 2019)
•17 models for cardiovascular disease in hypertensive population (Cai, 2020)
•14 models for ICU delirium risk (Chen, 2020)
•14 models for diabetic retinopathy progression (Haider, 2019)
•1382 models for cardiovascular disease (Wessler, 2021)
•731 models related to COVID-19 (Wynants, 2020)
•408 models for COPD prognosis (Bellou, 2019)
•363 models for cardiovascular disease general population (Damen, 2016)
•327 models for toxicity prediction after radiotherapy (Takada, 2022)
•263 prognosis models in obstetrics (Kleinrouweler, 2016)
•258 models mortality after general trauma (Munter, 2017)
•160 female-specific models for cardiovascular disease (Baart, 2019)
•142 models for mortality prediction in preterm infants (van Beek, 2021)
•119 models for critical care prognosis in LMIC (Haniffa, 2018)
•101 models for primary gastric cancer prognosis (Feng, 2019)
•99 models for neck pain (Wingbermühle, 2018)
•81 models for sudden cardiac arrest (Carrick, 2020)
•74 models for contrast-induced acute kidney injury (Allen, 2017)
•73 models for 28/30 day hospital readmission (Zhou, 2016)
•68 models for preeclampsia (De Kat, 2019)
•68 models for living donor kidney/iver transplant counselling (Haller, 2022)
•67 models for traumatic brain injury prognosis (Dijkland, 2019)
•64 models for suicide / suicide attempt (Belsher, 2019)
•61 models for dementia (Hou, 2019)
•58 modelsforbreastcancerprognosis(Phung, 2019)
•52 modelsforpre‐eclampsia(Townsend, 2019)
•52 modelsforcolorectalcancerrisk (Usher-Smith, 2016)
•48 modelsforincident hypertension(Sun, 2017)
•46 models for melanoma (Kaiser, 2020)
•46 models for prognosis after carotid revascularisation (Volkers, 2017)
•43 models for mortality in critically ill (Keuning, 2019)
Over 260 systematic reviews of clinical prediction models
Landscape of clinicalpredictionmodels
•42 models for kidney failure in chronic kidney disease (Ramspek, 2019)
•40 models for incident heart failure (Sahle, 2017)
•37 models for treatment response in pulmonary TB (Peetluk, 2021)
•35 models for in vitro fertilisation (Ratna, 2020)
•34 models for stroke in type-2 diabetes (Chowdhury, 2019)
•34 models for graft failure in kidney transplantation (Kabore, 2017)
•31 models for length of stay in ICU (Verburg, 2016)
•30 models for low back pain (Haskins, 2015)
•27 models for pediatric early warning systems (Trubey, 2019)
•27 models for malaria prognosis (Njim, 2019)
•26 models for postoperative outcomes colorectal cancer (Souwer, 2020)
•26 models for childhood asthma (Kothalawa, 2020)
•25 models for lung cancer risk (Gray, 2016)
•25 models for re-admissionafteradmittedforheartfailure (Mahajan, 2018)
•23 models for recovery after ischemic stroke (Jampathong, 2018)
•23 models for delirium in older adults (Lindroth, 2018)
•21 models for atrial fibrillation detection in community (Himmelreich, 2020)
•19 models for survival after resectable pancreatic cancer (Stijker, 2019)
•18 models for recurrence hep. carc. after liver transplant (Al-Ameri, 2020)
•18 models for future hypertension in children (Hamoen, 2018)
•18 models for risk of falls after stroke (Walsh, 2016)
•18 models for mortality in acute pancreatitis (Di, 2016)
•17 models for bacterial meningitis (van Zeggeren, 2019)
•17 models for cardiovascular disease in hypertensive population (Cai, 2020)
•14 models for ICU delirium risk (Chen, 2020)
•14 models for diabetic retinopathy progression (Haider, 2019)
•1382 models for cardiovascular disease (Wessler, 2021)
•731 models related to COVID-19 (Wynants, 2020)
•408 models for COPD prognosis (Bellou, 2019)
•363 models for cardiovascular disease general population (Damen, 2016)
•327 models for toxicity prediction after radiotherapy (Takada, 2022)
•263 prognosis models in obstetrics (Kleinrouweler, 2016)
•258 models mortality after general trauma (Munter, 2017)
•160 female-specific models for cardiovascular disease (Baart, 2019)
•142 models for mortality prediction in preterm infants (van Beek, 2021)
•119 models for critical care prognosis in LMIC (Haniffa, 2018)
•101 models for primary gastric cancer prognosis (Feng, 2019)
•99 models for neck pain (Wingbermühle, 2018)
•81 models for sudden cardiac arrest (Carrick, 2020)
•74 models for contrast-induced acute kidney injury (Allen, 2017)
•73 models for 28/30 day hospital readmission (Zhou, 2016)
•68 models for preeclampsia (De Kat, 2019)
•68 models for living donor kidney/iver transplant counselling (Haller, 2022)
•67 models for traumatic brain injury prognosis (Dijkland, 2019)
•64 models for suicide / suicide attempt (Belsher, 2019)
•61 models for dementia (Hou, 2019)
•58 modelsforbreastcancerprognosis(Phung, 2019)
•52 modelsforpre‐eclampsia(Townsend, 2019)
•52 modelsforcolorectalcancerrisk (Usher-Smith, 2016)
•48 modelsforincident hypertension(Sun, 2017)
•46 models for melanoma (Kaiser, 2020)
•46 models for prognosis after carotid revascularisation (Volkers, 2017)
•43 models for mortality in critically ill (Keuning, 2019)
Over 260 systematic reviews of clinical prediction models
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Clinical prediction models
•> 150,000 clinical prediction models exist
•From simple scoring rules (e.g. APGAR) to increasingly complex
AI-based prediction models
Source: Arshiat al 2024, OSF, doi: 10.31219/osf.io/4txc6 .
Clinical prediction models
•> 150,000 clinical prediction models exist
•From simple scoring rules (e.g. APGAR) to increasingly complex
AI-based prediction models
Source: Arshiat al 2024, OSF, doi: 10.31219/osf.io/4txc6 .
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
A new clinical prediction model
is developed
every 1.5 hours
Source: Arshiat al 2024, OSF, doi: 10.31219/osf.io/4txc6 .
is developed
every 1.5 hours
Source: Arshiat al 2024, OSF, doi: 10.31219/osf.io/4txc6 .
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
PREDICTION MODELS USED IN PRACTICE
PREDICTION MODELS THAT WILL NEVER BE USED IN PRACTICE
RESEARCH WASTE?
PREDICTION MODELS USED IN PRACTICE
PREDICTION MODELS THAT WILL NEVER BE USED IN PRACTICE
RESEARCH WASTE?
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Example: living review
COVID-19 predictionmodels
•731 prediction models between
March 2020 and February 2021
•Many models poorly reported
•Only 4% low risk of bias
Example: living review
COVID-19 predictionmodels
•731 prediction models between
March 2020 and February 2021
•Many models poorly reported
•Only 4% low risk of bias
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Externalvalidation
COVID-19 prediction
models
Externalvalidation
COVID-19 prediction
models
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Not just COVID
Not just COVID
What is AI going to do to the
field of clinical prediction
models?
field of clinical prediction
models?
What is AI doing for us now?
Self driving cars, etc
CreatedusingDall-E
CreatedusingDall-E
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
IBM Watson winning Jeopardy! (2011)
https://bbc.in/2TMvV8I
IBM Watson winning Jeopardy! (2011)
https://bbc.in/2TMvV8I
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
IBM Watson for oncology
bit.ly/2LxiWGj ; bit.ly/3Esu68T
IBM Watson for oncology
bit.ly/2LxiWGj ; bit.ly/3Esu68T
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Tech company business model
Tech company business model
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Tech company business model
https://bit.ly/2HSp8X5; https://bit.ly/2Z0Pfop; https://bit.ly/2KIcpHG; https://bit.ly/33IJhr9
Tech company business model
https://bit.ly/2HSp8X5; https://bit.ly/2Z0Pfop; https://bit.ly/2KIcpHG; https://bit.ly/33IJhr9
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Proportionof studies indexedin MedlinewiththeMedicalSubject
Heading(MeSH) term “ArtificialIntelligence”
Faeset al. doi: 10.3389/fdgth.2022.833912
Proportionof studies indexedin MedlinewiththeMedicalSubject
Heading(MeSH) term “ArtificialIntelligence”
Faeset al. doi: 10.3389/fdgth.2022.833912
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Other success stories
https://go.nature.com/2VG2hS7; https://bbc.in/2Z1drXQ; https://bit.ly/2TAfRIP
Other success stories
https://go.nature.com/2VG2hS7; https://bbc.in/2Z1drXQ; https://bit.ly/2TAfRIP
https://twitter.com/AndrewLBeam/status/1620855064033382401?s=20&t=VO9_LdFFCj_wcwIQLvKcIQ
Source: Ilse Kant (UMC Utrecht)
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmedenhttps://bit.ly/2v2aokk
Kopenhagen, 22 Aug 2024 @MaartenvSmedenAyers, JAMA Int Med, 2023, doi: 10.1001/jamainternmed.2023.1838
*Answers by healthcare professionals on Redit vs ChatGPT
*Answers by healthcare professionals on Redit vs ChatGPT
Kopenhagen, 22 Aug 2024 @MaartenvSmedenSource: https://twitter.com/TansuYegen/status/1635388676539813889?s=20
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Source: https://www.science.org/content/article/alarmed-tech-leaders-call-ai-research-pause
Source: https://www.science.org/content/article/alarmed-tech-leaders-call-ai-research-pause
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Reviewer#2
Reviewer#2
Three Myths
about
Machine
learning
about
Machine
learning
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Myth 1: “ML methods come from computer science”
Leo Breiman Jerome H
Friedman
Trevor Hastie Robert TibshiraniDanielaWitten
CART, random forest Gradientboosting Elements of statistical
learning
Lasso Introductiontostatistical
learning
Edu Physics/Math Physics Statistics Statistics Statistics
Job titleProfessor of StatisticsProfessor of StatisticsProfessor of StatisticsProfessor of StatisticsProfessor of Statistics
Myth 1: “ML methods come from computer science”
Leo Breiman Jerome H
Friedman
Trevor Hastie Robert TibshiraniDanielaWitten
CART, random forest Gradientboosting Elements of statistical
learning
Lasso Introductiontostatistical
learning
Edu Physics/Math Physics Statistics Statistics Statistics
Job titleProfessor of StatisticsProfessor of StatisticsProfessor of StatisticsProfessor of StatisticsProfessor of Statistics
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Myth 2:“ML methods are for prediction, statistics is
for explaining”
1
See further: KreiffandDiaz Ordaz; https://bit.ly/2m1eYdK
ML and causal inference, small selection
1
•Superlearner (e.g. van der Laan)
•High dimensional propensity scores (e.g. Schneeweiss)
•Causal forests (e.g. Athey)
•The book of why (Pearl)
Myth 2:“ML methods are for prediction, statistics is
for explaining”
1
See further: KreiffandDiaz Ordaz; https://bit.ly/2m1eYdK
ML and causal inference, small selection
1
•Superlearner (e.g. van der Laan)
•High dimensional propensity scores (e.g. Schneeweiss)
•Causal forests (e.g. Athey)
•The book of why (Pearl)
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Twocultures
Breiman, Stat Sci, 2001, DOI: 10.1214/ss/1009213726
Twocultures
Breiman, Stat Sci, 2001, DOI: 10.1214/ss/1009213726
Kopenhagen, 22 Aug 2024 @MaartenvSmedenFaeset al. doi: 1 0.3 389/fd gth .20 22.833912
Language
Language
Kopenhagen, 22 Aug 2024 @MaartenvSmedenRobert Tibshirani: https://stanford.io/2zqEGfr
Machine learning: large grant = $1,000,000
Statistics: large grant = $50,000
Machine learning: large grant = $1,000,000
Statistics: large grant = $50,000
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
ML referstoa culture, nottomethods
Distinguishing between statistics and machine learning
•Substantial overlap methods used by both cultures
•Substantial overlap analysis goals
•Attempts to separate the two frequently result in disagreement
Pragmatic approach:
I’ll use “ML” to refer to models roughly outside of the traditional regression
types of analysis: decision trees (and descendants), SVMs, neural networks
(including Deep learning), boosting etc.
ML referstoa culture, nottomethods
Distinguishing between statistics and machine learning
•Substantial overlap methods used by both cultures
•Substantial overlap analysis goals
•Attempts to separate the two frequently result in disagreement
Pragmatic approach:
I’ll use “ML” to refer to models roughly outside of the traditional regression
types of analysis: decision trees (and descendants), SVMs, neural networks
(including Deep learning), boosting etc.
Kopenhagen, 22 Aug 2024 @MaartenvSmedendoi: 10.1001/jamapediatrics.2023.0034
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Myth 3: Machine learning is (always) better at
prediction
Christodoulouet al. Journal of ClinicalEpidemiology, 2019, doi: 10.1016/j.jclinepi.2019.02.004
Myth 3: Machine learning is (always) better at
prediction
Christodoulouet al. Journal of ClinicalEpidemiology, 2019, doi: 10.1016/j.jclinepi.2019.02.004
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Myth 3: Machine learning is (always) better at
prediction
Christodoulouet al. Journal of ClinicalEpidemiology, 2019, doi: 10.1016/j.jclinepi.2019.02.004
Myth 3: Machine learning is (always) better at
prediction
Christodoulouet al. Journal of ClinicalEpidemiology, 2019, doi: 10.1016/j.jclinepi.2019.02.004
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Sources of predictionerror
Y=????????????+??????
For a model ??????theexpectedtest predictionerror is:
σ
2
+bias
2መ??????
????????????+varመ??????
????????????
See equation2.46 in Hastieet al., theelementsof statisticallearning, https://stanford.io/2voWjra
Irreducible error Meansquaredpredictionerror
(withE??????=0,var??????=??????
2
,valuesin??????arenotrandom)
What we don’t model How we model
≈≈
Sources of predictionerror
Y=????????????+??????
For a model ??????theexpectedtest predictionerror is:
σ
2
+bias
2መ??????
????????????+varመ??????
????????????
See equation2.46 in Hastieet al., theelementsof statisticallearning, https://stanford.io/2voWjra
Irreducible error Meansquaredpredictionerror
(withE??????=0,var??????=??????
2
,valuesin??????arenotrandom)
What we don’t model How we model
≈≈
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Sources of predictionerror
Y=????????????+??????
For a model ??????theexpectedtest predictionerror is:
σ
2
+bias
2መ??????
????????????+varመ??????
????????????
See equation2.46 in Hastieet al., theelementsof statisticallearning, https://stanford.io/2voWjra
Irreducible error Meansquaredpredictionerror
(withE??????=0,var??????=??????
2
,valuesin??????arenotrandom)
What we don’t model How we model
≈≈
In words, two main components for error in predictions are:
•Mean squared predictor error
•Under control of the modeler
Sources of predictionerror
Y=????????????+??????
For a model ??????theexpectedtest predictionerror is:
σ
2
+bias
2መ??????
????????????+varመ??????
????????????
See equation2.46 in Hastieet al., theelementsof statisticallearning, https://stanford.io/2voWjra
Irreducible error Meansquaredpredictionerror
(withE??????=0,var??????=??????
2
,valuesin??????arenotrandom)
What we don’t model How we model
≈≈
In words, two main components for error in predictions are:
•Mean squared predictor error
•Under control of the modeler
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Sources of predictionerror
Y=????????????+??????
For a model ??????theexpectedtest predictionerror is:
σ
2
+bias
2መ??????
????????????+varመ??????
????????????
See equation2.46 in Hastieet al., theelementsof statisticallearning, https://stanford.io/2voWjra
Irreducible error Meansquaredpredictionerror
(withE??????=0,var??????=??????
2
,valuesin??????arenotrandom)
What we don’t model How we model
≈≈
In words, two main components for error in predictions are:
•Mean squared predictor error
•Under control of the modeler
overfitting underfitting ”just right”
Sources of predictionerror
Y=????????????+??????
For a model ??????theexpectedtest predictionerror is:
σ
2
+bias
2መ??????
????????????+varመ??????
????????????
See equation2.46 in Hastieet al., theelementsof statisticallearning, https://stanford.io/2voWjra
Irreducible error Meansquaredpredictionerror
(withE??????=0,var??????=??????
2
,valuesin??????arenotrandom)
What we don’t model How we model
≈≈
In words, two main components for error in predictions are:
•Mean squared predictor error
•Under control of the modeler
overfitting underfitting ”just right”
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Sources of predictionerror
Y=????????????+??????
For a model ??????theexpectedtest predictionerror is:
σ
2
+bias
2መ??????
????????????+varመ??????
????????????
See equation2.46 in Hastieet al., theelementsof statisticallearning, https://stanford.io/2voWjra
Irreducible error Meansquaredpredictionerror
(withE??????=0,var??????=??????
2
,valuesin??????arenotrandom)
What we don’t model How we model
≈≈
In words, two main components for error in predictions are:
•Mean squared predictor error
•Under control of the modeler
•Irreducible error
•Not under direct control of the modeler
Sources of predictionerror
Y=????????????+??????
For a model ??????theexpectedtest predictionerror is:
σ
2
+bias
2መ??????
????????????+varመ??????
????????????
See equation2.46 in Hastieet al., theelementsof statisticallearning, https://stanford.io/2voWjra
Irreducible error Meansquaredpredictionerror
(withE??????=0,var??????=??????
2
,valuesin??????arenotrandom)
What we don’t model How we model
≈≈
In words, two main components for error in predictions are:
•Mean squared predictor error
•Under control of the modeler
•Irreducible error
•Not under direct control of the modeler
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
What can we do to reduce “irreducible” error?
Changingtheinformation
•Using text(NLP/textmining)
•For research: e.g. predictinglife expectancy
https://bit.ly/2k8Ao8e
•Analyzingsocialmedia posts
•e.g. pharmacovigilance, adverse events monitoring via Twitter posts
https://bit.ly/2m0KKrg
•Speech signalprocessing
•e.g. Parkinson‟sdisease,
https://bit.ly/2v3ZdHR
•Medicalimaging
What can we do to reduce “irreducible” error?
Changingtheinformation
•Using text(NLP/textmining)
•For research: e.g. predictinglife expectancy
https://bit.ly/2k8Ao8e
•Analyzingsocialmedia posts
•e.g. pharmacovigilance, adverse events monitoring via Twitter posts
https://bit.ly/2m0KKrg
•Speech signalprocessing
•e.g. Parkinson‟sdisease,
https://bit.ly/2v3ZdHR
•Medicalimaging
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Examples where
AI has done well
Examples where
AI has done well
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Example: retinaldisease
Gulshanet al, JAMA, 2016, 10.1001/jama.2016.17216; Picture retinopathy: https://bit.ly/2kB3X2w
Diabeticretinopathy
Deep learning (= Neural network)
•128,000 images
•Transfer learning(preinitialization)
•Sensitivityandspecificity> .90
•Estimatedfromtraining data
Example: retinaldisease
Gulshanet al, JAMA, 2016, 10.1001/jama.2016.17216; Picture retinopathy: https://bit.ly/2kB3X2w
Diabeticretinopathy
Deep learning (= Neural network)
•128,000 images
•Transfer learning(preinitialization)
•Sensitivityandspecificity> .90
•Estimatedfromtraining data
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Approval of AI devices by FDA rapidly growing
Source: https://tinyurl.com/khn4dvyb (accessed21/08/2024)
Approval of AI devices by FDA rapidly growing
Source: https://tinyurl.com/khn4dvyb (accessed21/08/2024)
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Examples where
AI has done poorly
Examples where
AI has done poorly
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Predictingmortality–theconclusion
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344
Predictingmortality–theconclusion
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Predictingmortality–theresults
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344
Predictingmortality–theresults
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Predictingmortality–themedia
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344; https://bit.ly/2Q6H41R; https://bit.ly/2m3RLrn
Predictingmortality–themedia
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344; https://bit.ly/2Q6H41R; https://bit.ly/2m3RLrn
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
HYPE!
HYPE!
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
RecidivismAlgorithm
Pro-publica(2016) https://bit.ly/1XMKh5R
RecidivismAlgorithm
Pro-publica(2016) https://bit.ly/1XMKh5R
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Skin cancerandrulers
Estevaet al., Nature, 2016, DOI: 10.1038/nature21056; https://bit.ly/2lE0vV0
Skin cancerandrulers
Estevaet al., Nature, 2016, DOI: 10.1038/nature21056; https://bit.ly/2lE0vV0
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmedenhttps://www.tctmd.com/news/machine-learning-helps-predict-hospital-mortality-post-tavr-skepticism-abounds
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
AI assistance leads tomoreaccurate diagnosis of livercancer!
AI assistance leads tomoreaccurate diagnosis of livercancer!
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
AI assistance leads tomoreaccurate diagnosis of livercancer! IfAI is correct
AI assistance leads tolessaccurate diagnosis of livercancer! IfAI is incorrect
AI assistance leads tomoreaccurate diagnosis of livercancer! IfAI is correct
AI assistance leads tolessaccurate diagnosis of livercancer! IfAI is incorrect
How can the field of clinical prediction
models using AI maximise benefits
and minimize risks and waste?
models using AI maximise benefits
and minimize risks and waste?
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Image source: http: //www.meditationcircle.org.uk/notes/acceptance/
Image source: http: //www.meditationcircle.org.uk/notes/acceptance/
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
The ML/AI model is only one small element in
getting the model in clinical practice
Source: https://tinyurl.com/jr23pdsk; courtesyDrIlse Kant (UMCU)
The ML/AI model is only one small element in
getting the model in clinical practice
Source: https://tinyurl.com/jr23pdsk; courtesyDrIlse Kant (UMCU)
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Leakypipeline of clinicalpredictionmodels
Van Royen et al, ERJ, doi: 10.1183/13993003.00250-2022, alsocreditstoLaure Wynants
Leakypipeline of clinicalpredictionmodels
Van Royen et al, ERJ, doi: 10.1183/13993003.00250-2022, alsocreditstoLaure Wynants
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Flexiblealgorithmsare data hungry
Fromslide deck Ben van Calster: https://bit.ly/38Aqmjs
Flexiblealgorithmsare data hungry
Fromslide deck Ben van Calster: https://bit.ly/38Aqmjs
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Flexiblealgorithmsare energy hungry
The costsof training (cloudcomputing) theTransformer
once(!) are estimatedat 1 to3 millionDollars
https://bit.ly/33Dj38X
Flexiblealgorithmsare energy hungry
The costsof training (cloudcomputing) theTransformer
once(!) are estimatedat 1 to3 millionDollars
https://bit.ly/33Dj38X
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Expect heterogeneity in model performance
Wessler, CirculationCQO ,2021, doi:10.1161/CIRCOUTCOMES.121.007858
Expect heterogeneity in model performance
Wessler, CirculationCQO ,2021, doi:10.1161/CIRCOUTCOMES.121.007858
€
kr
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Dutch guideline predictionmodelsbasedof AI
https://www.leidraad-ai.nl/
Dutch guideline predictionmodelsbasedof AI
https://www.leidraad-ai.nl/
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Dutch guideline predictionmodelsbasedof AI
https://www.leidraad-ai.nl/Collection and
management of the
data
Phase 1
Development of the
AIP
Phase 2
Validation of the
AIPA
Phase 3
Development of the
required software
Phase 4
Impact assessment
of the AIPA in
combination with
the software
Phase 5
Implementation
and use of the AIPA
with software in
daily practice
Phase 6
Saskia Haitjema
Andre Dekker
Paul Algra
Amy Eikelenboom
Christian van
Ginkel
Martine de Vries
Daniel Oberski
Desy Kakiay
Kicky van
Leeuwen
Joran Lokkerbol
Evangelos
Kanoulas
Gabrielle
Davelaar
Wouter Veldhuis
Bart-Jan Verhoeff
Vincent Stirler
Daanvan den
Donk
HuibBurger
Giovanni Cina
Martijn van der
Meulen
MauritsKaptein
Floor van
Leeuwen
Eggevan der Poel
Marcel Hilgersom
Sade Faneyte
Jonas Teuwen
Teus Kappen
Ewout Steyerberg
Leo Hovestadt
René Drost
Bart Geerts
Anne de Hond
René Verhaart
NynkeBreimer
Karen Wiegant
Laure Wynants
Lysette
Meuleman
Dutch guideline predictionmodelsbasedof AI
https://www.leidraad-ai.nl/Collection and
management of the
data
Phase 1
Development of the
AIP
Phase 2
Validation of the
AIPA
Phase 3
Development of the
required software
Phase 4
Impact assessment
of the AIPA in
combination with
the software
Phase 5
Implementation
and use of the AIPA
with software in
daily practice
Phase 6
Saskia Haitjema
Andre Dekker
Paul Algra
Amy Eikelenboom
Christian van
Ginkel
Martine de Vries
Daniel Oberski
Desy Kakiay
Kicky van
Leeuwen
Joran Lokkerbol
Evangelos
Kanoulas
Gabrielle
Davelaar
Wouter Veldhuis
Bart-Jan Verhoeff
Vincent Stirler
Daanvan den
Donk
HuibBurger
Giovanni Cina
Martijn van der
Meulen
MauritsKaptein
Floor van
Leeuwen
Eggevan der Poel
Marcel Hilgersom
Sade Faneyte
Jonas Teuwen
Teus Kappen
Ewout Steyerberg
Leo Hovestadt
René Drost
Bart Geerts
Anne de Hond
René Verhaart
NynkeBreimer
Karen Wiegant
Laure Wynants
Lysette
Meuleman
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
AI ecosystem in the University Medical Center Utrecht
You are here
AI ecosystem in the University Medical Center Utrecht
You are here
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
R&D concentrated in 5 AI labs
https://www.umcutrecht.nl/en/campaign/ai-labs
R&D concentrated in 5 AI labs
https://www.umcutrecht.nl/en/campaign/ai-labs
Team science
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
•Hype
•AI rebranding and
reinventions
•Traditional issues such
as low N, lack of
validation, poor
reporting, data quality,
generalizability
•More research waste
•Energy consumption
•Other expenses beyond
model training
AI BLESSINGS AND CURSES
•Real innovation
•Methods/architectures
allowing (unstructured)
use of new types of
data at scale
•Computing power
•Software
•Clinical trials showing
benefit of AI assistance
•Willingness to invest in
prediction using AI
•Hype
•AI rebranding and
reinventions
•Traditional issues such
as low N, lack of
validation, poor
reporting, data quality,
generalizability
•More research waste
•Energy consumption
•Other expenses beyond
model training
AI BLESSINGS AND CURSES
•Real innovation
•Methods/architectures
allowing (unstructured)
use of new types of
data at scale
•Computing power
•Software
•Clinical trials showing
benefit of AI assistance
•Willingness to invest in
prediction using AI
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Maarten van Smeden
Julius Center for Health Sciences and Primary Care
University Medical Center Utrecht
Director of UMC Utrecht AI methods lab
Team lead of health data science group
Head of Julius Center’s methods program
E-mail: [email protected]
Maarten van Smeden
Julius Center for Health Sciences and Primary Care
University Medical Center Utrecht
Director of UMC Utrecht AI methods lab
Team lead of health data science group
Head of Julius Center’s methods program
E-mail: [email protected]
Kopenhagen, 22 Aug 2024 @MaartenvSmeden
Download
Download Slideshow Get the original presentation fileQuick Actions
Statistics
- Views 777
- Slides 107
- Age 468 days
Related Slideshows
36
Clinical approach Dyspnoea A simple and practical approach.pptx
ShajahanPS
25 views
238
Cancer Awareness therapy for public by Dr Kanhu Charan Patro
kanhucpatro
24 views
41
Prof Satyadas Memorial oration Kozhikode.pptx
ShajahanPS
20 views
26
Viral Conjunctivitis and it;s managment.pptx
ZaidAzhar
34 views
30
Essential Thrombocythemia 15 Years of Experience at the Hematology Department, Algies, Algeria.pdf
sbelakehal
30 views
10
Hypertension sign symptoms cmdt style with regime
androiddabest
31 views