Super Learning in Mathematical Algorithms
razigineer
17 views
181 slides
Sep 16, 2025
Slide 1 of 181
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
About This Presentation
Super Learning Algorithm
Slide Content
Introductionto
SuperLearning
Ted Westling, PhD
Postdoctoral Researcher
Center for Causal Inference
Perelman School of Medicine
University of Pennsylvania
September 25, 2018
1 / 48
SuperLearning
Ted Westling, PhD
Postdoctoral Researcher
Center for Causal Inference
Perelman School of Medicine
University of Pennsylvania
September 25, 2018
1 / 48
Learning Goals
Conceptual understanding of Super Learning (SL)
Comfort with the SuperLearner R packageAwareness of the mathematical backbone of SL
2 / 48
Conceptual understanding of Super Learning (SL)
Comfort with the SuperLearner R packageAwareness of the mathematical backbone of SL
2 / 48
Learning Goals
Conceptual understanding of Super Learning (SL)
Comfort with the SuperLearner R packageAwareness of the mathematical backbone of SL
2 / 48
Conceptual understanding of Super Learning (SL)
Comfort with the SuperLearner R packageAwareness of the mathematical backbone of SL
2 / 48
Learning Goals
Conceptual understanding of Super Learning (SL)
Comfort with the SuperLearner R packageAwareness of the mathematical backbone of SL
2 / 48
Conceptual understanding of Super Learning (SL)
Comfort with the SuperLearner R packageAwareness of the mathematical backbone of SL
2 / 48
Outline
I.Motivation and description of SL (30 minutes)
II.Lab 1: Vanilla SL for a continuous outcome (30 minutes)
III.Mathematical presentation of SL (20 minutes)
IV.Lab 2: Vanilla SL for a binary outcome (30 minutes)
15 minute break
3 / 48
I.Motivation and description of SL (30 minutes)
II.Lab 1: Vanilla SL for a continuous outcome (30 minutes)
III.Mathematical presentation of SL (20 minutes)
IV.Lab 2: Vanilla SL for a binary outcome (30 minutes)
15 minute break
3 / 48
Outline
15 minute break
V.Bells and whistles: Screens, weights, and CV-SL (30
minutes)
VI.Lab 3: Binary outcome redux (40 minutes)
VII.Lab 4: Case-control analysis of Fluzone vaccine (30
minutes)
4 / 48
15 minute break
V.Bells and whistles: Screens, weights, and CV-SL (30
minutes)
VI.Lab 3: Binary outcome redux (40 minutes)
VII.Lab 4: Case-control analysis of Fluzone vaccine (30
minutes)
4 / 48
I.Motivationand
descriptionofSuper
Learning
4 / 48
descriptionofSuper
Learning
4 / 48
Notation
Yis a univariate outcome
Xis ap-variate set of predictorsWe observenindependent copies
(Y1;X1); : : : ;(Yn;Xn)
from the joint distribution of(Y;X).
5 / 48
Yis a univariate outcome
Xis ap-variate set of predictorsWe observenindependent copies
(Y1;X1); : : : ;(Yn;Xn)
from the joint distribution of(Y;X).
5 / 48
Notation
Yis a univariate outcome
Xis ap-variate set of predictorsWe observenindependent copies
(Y1;X1); : : : ;(Yn;Xn)
from the joint distribution of(Y;X).
5 / 48
Yis a univariate outcome
Xis ap-variate set of predictorsWe observenindependent copies
(Y1;X1); : : : ;(Yn;Xn)
from the joint distribution of(Y;X).
5 / 48
Notation
Yis a univariate outcome
Xis ap-variate set of predictorsWe observenindependent copies
(Y1;X1); : : : ;(Yn;Xn)
from the joint distribution of(Y;X).
5 / 48
Yis a univariate outcome
Xis ap-variate set of predictorsWe observenindependent copies
(Y1;X1); : : : ;(Yn;Xn)
from the joint distribution of(Y;X).
5 / 48
The problem
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
The problem
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
The problem
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
The problem
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
The problem
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
The problem
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
The problem
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
We want to estimate a function, e.g.:
Super Learning can be applied in all of the above settingsWe will focus on estimating the regression function
(x) :=E[YjX=x].
6 / 48
Why?
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
Why?
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
Why?
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
Why?
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
Why?
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
Why?
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
Why?
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
1.Exploratory analysis
2.Imputationof missing values3.Predictionfor new observations4. prediction quality/comparingcompeting
estimators
5. nuisance parameterestimator6.Conrmatory analysis/hypothesis testing
(not our goal here)
7 / 48
We want to estimate(x) =E[YjX=x].
How should we do it?
8 / 48
How should we do it?
8 / 48
We want to estimate(x) =E[YjX=x].
How should we do it?
8 / 48
How should we do it?
8 / 48
We want to estimate(x) =E[YjX=x].
How should we do it?GAM
8 / 48
How should we do it?GAM
8 / 48
We want to estimate(x) =E[YjX=x].
How should we do it?GAM
Random Forest
8 / 48
How should we do it?GAM
Random Forest
8 / 48
We want to estimate(x) =E[YjX=x].
How should we do it?GAM
Random Forest
Neural network
8 / 48
How should we do it?GAM
Random Forest
Neural network
8 / 48
We want to estimate(x) =E[YjX=x].
How should we do it?GAM
Random Forest
Neural network
GLM
8 / 48
How should we do it?GAM
Random Forest
Neural network
GLM
8 / 48
How do we choose which algorithm to use?
9 / 48
9 / 48
Super Learning is:
Anensemble methodfor combining
predictions from many candidate machine
learning algorithms
10 / 48
Anensemble methodfor combining
predictions from many candidate machine
learning algorithms
10 / 48
Measuring algorithm
performance
Suppose^1; : : : ;^
Kare candidate estimators of.
kwill always indexestimators, andiwill always index
observations(e.g. study participants)
Themean squared errorof^
k,
MSE(^
k) =E
h
(Y^
k(X))
2
i
measures the performance of^
kas an estimator of.
If we knewMSE(^
k), we could choose the^
kwith the
smallestMSE(^
k).
11 / 48
performance
Suppose^1; : : : ;^
Kare candidate estimators of.
kwill always indexestimators, andiwill always index
observations(e.g. study participants)
Themean squared errorof^
k,
MSE(^
k) =E
h
(Y^
k(X))
2
i
measures the performance of^
kas an estimator of.
If we knewMSE(^
k), we could choose the^
kwith the
smallestMSE(^
k).
11 / 48
Measuring algorithm
performance
Suppose^1; : : : ;^
Kare candidate estimators of.
kwill always indexestimators, andiwill always index
observations(e.g. study participants)
Themean squared errorof^
k,
MSE(^
k) =E
h
(Y^
k(X))
2
i
measures the performance of^
kas an estimator of.
If we knewMSE(^
k), we could choose the^
kwith the
smallestMSE(^
k).
11 / 48
performance
Suppose^1; : : : ;^
Kare candidate estimators of.
kwill always indexestimators, andiwill always index
observations(e.g. study participants)
Themean squared errorof^
k,
MSE(^
k) =E
h
(Y^
k(X))
2
i
measures the performance of^
kas an estimator of.
If we knewMSE(^
k), we could choose the^
kwith the
smallestMSE(^
k).
11 / 48
Measuring algorithm
performance
Suppose^1; : : : ;^
Kare candidate estimators of.
kwill always indexestimators, andiwill always index
observations(e.g. study participants)
Themean squared errorof^
k,
MSE(^
k) =E
h
(Y^
k(X))
2
i
measures the performance of^
kas an estimator of.
If we knewMSE(^
k), we could choose the^
kwith the
smallestMSE(^
k).
11 / 48
performance
Suppose^1; : : : ;^
Kare candidate estimators of.
kwill always indexestimators, andiwill always index
observations(e.g. study participants)
Themean squared errorof^
k,
MSE(^
k) =E
h
(Y^
k(X))
2
i
measures the performance of^
kas an estimator of.
If we knewMSE(^
k), we could choose the^
kwith the
smallestMSE(^
k).
11 / 48
Measuring algorithm
performance
Suppose^1; : : : ;^
Kare candidate estimators of.
kwill always indexestimators, andiwill always index
observations(e.g. study participants)
Themean squared errorof^
k,
MSE(^
k) =E
h
(Y^
k(X))
2
i
measures the performance of^
kas an estimator of.
If we knewMSE(^
k), we could choose the^
kwith the
smallestMSE(^
k).
11 / 48
performance
Suppose^1; : : : ;^
Kare candidate estimators of.
kwill always indexestimators, andiwill always index
observations(e.g. study participants)
Themean squared errorof^
k,
MSE(^
k) =E
h
(Y^
k(X))
2
i
measures the performance of^
kas an estimator of.
If we knewMSE(^
k), we could choose the^
kwith the
smallestMSE(^
k).
11 / 48
Estimating MSE
MSE(^
k) =E
h
(Y^
k(X))
2
i
It is tempting to take
[
MSE(^
k) =
1
n
P
n
i=1
[Y
i^
k(X
i)]
2
.
This estimator will favor^
kwhich areovert, because^
k
are trained on the same data used to evaluate the MSE.
Analogy: a student has the exam questionsbeforetaking
the exam!
Instead, we estimate MSE usingcross-validation.
12 / 48
MSE(^
k) =E
h
(Y^
k(X))
2
i
It is tempting to take
[
MSE(^
k) =
1
n
P
n
i=1
[Y
i^
k(X
i)]
2
.
This estimator will favor^
kwhich areovert, because^
k
are trained on the same data used to evaluate the MSE.
Analogy: a student has the exam questionsbeforetaking
the exam!
Instead, we estimate MSE usingcross-validation.
12 / 48
Estimating MSE
MSE(^
k) =E
h
(Y^
k(X))
2
i
It is tempting to take
[
MSE(^
k) =
1
n
P
n
i=1
[Y
i^
k(X
i)]
2
.
This estimator will favor^
kwhich areovert, because^
k
are trained on the same data used to evaluate the MSE.
Analogy: a student has the exam questionsbeforetaking
the exam!
Instead, we estimate MSE usingcross-validation.
12 / 48
MSE(^
k) =E
h
(Y^
k(X))
2
i
It is tempting to take
[
MSE(^
k) =
1
n
P
n
i=1
[Y
i^
k(X
i)]
2
.
This estimator will favor^
kwhich areovert, because^
k
are trained on the same data used to evaluate the MSE.
Analogy: a student has the exam questionsbeforetaking
the exam!
Instead, we estimate MSE usingcross-validation.
12 / 48
Estimating MSE
MSE(^
k) =E
h
(Y^
k(X))
2
i
It is tempting to take
[
MSE(^
k) =
1
n
P
n
i=1
[Y
i^
k(X
i)]
2
.
This estimator will favor^
kwhich areovert, because^
k
are trained on the same data used to evaluate the MSE.
Analogy: a student has the exam questionsbeforetaking
the exam!
Instead, we estimate MSE usingcross-validation.
12 / 48
MSE(^
k) =E
h
(Y^
k(X))
2
i
It is tempting to take
[
MSE(^
k) =
1
n
P
n
i=1
[Y
i^
k(X
i)]
2
.
This estimator will favor^
kwhich areovert, because^
k
are trained on the same data used to evaluate the MSE.
Analogy: a student has the exam questionsbeforetaking
the exam!
Instead, we estimate MSE usingcross-validation.
12 / 48
Estimating MSE
MSE(^
k) =E
h
(Y^
k(X))
2
i
It is tempting to take
[
MSE(^
k) =
1
n
P
n
i=1
[Y
i^
k(X
i)]
2
.
This estimator will favor^
kwhich areovert, because^
k
are trained on the same data used to evaluate the MSE.
Analogy: a student has the exam questionsbeforetaking
the exam!
Instead, we estimate MSE usingcross-validation.
12 / 48
MSE(^
k) =E
h
(Y^
k(X))
2
i
It is tempting to take
[
MSE(^
k) =
1
n
P
n
i=1
[Y
i^
k(X
i)]
2
.
This estimator will favor^
kwhich areovert, because^
k
are trained on the same data used to evaluate the MSE.
Analogy: a student has the exam questionsbeforetaking
the exam!
Instead, we estimate MSE usingcross-validation.
12 / 48
Estimating MSE
MSE(^
k) =E
h
(Y^
k(X))
2
i
It is tempting to take
[
MSE(^
k) =
1
n
P
n
i=1
[Y
i^
k(X
i)]
2
.
This estimator will favor^
kwhich areovert, because^
k
are trained on the same data used to evaluate the MSE.
Analogy: a student has the exam questionsbeforetaking
the exam!
Instead, we estimate MSE usingcross-validation.
12 / 48
MSE(^
k) =E
h
(Y^
k(X))
2
i
It is tempting to take
[
MSE(^
k) =
1
n
P
n
i=1
[Y
i^
k(X
i)]
2
.
This estimator will favor^
kwhich areovert, because^
k
are trained on the same data used to evaluate the MSE.
Analogy: a student has the exam questionsbeforetaking
the exam!
Instead, we estimate MSE usingcross-validation.
12 / 48
Cross-validation
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
13 / 48
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
13 / 48
Cross-validation
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
13 / 48
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
13 / 48
123456789101Fold 1123456789102Fold 2123456789103Fold 3123456789104Fold 4123456789105Fold 5123456789106Fold 6123456789107Fold 7123456789108Fold 8123456789109Fold 91234567891010Fold 10
Schematic of 10-fold cross-validation. Gray: training sets. Yellow: validation sets.
14 / 48
Schematic of 10-fold cross-validation. Gray: training sets. Yellow: validation sets.
14 / 48
Cross-validation
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
we obtain^
k;vusing thetraining set;we obtain^
k;v(X
i)forX
iin thevalidation setVv.3. cross-validated MSEis
[
MSE
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
[Y
i^
k;v(X
i)]
2
:
We average the MSEs of theVvalidation sets. 15 / 48
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
we obtain^
k;vusing thetraining set;we obtain^
k;v(X
i)forX
iin thevalidation setVv.3. cross-validated MSEis
[
MSE
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
[Y
i^
k;v(X
i)]
2
:
We average the MSEs of theVvalidation sets. 15 / 48
Cross-validation
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
we obtain^
k;vusing thetraining set;we obtain^
k;v(X
i)forX
iin thevalidation setVv.3. cross-validated MSEis
[
MSE
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
[Y
i^
k;v(X
i)]
2
:
We average the MSEs of theVvalidation sets. 15 / 48
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
we obtain^
k;vusing thetraining set;we obtain^
k;v(X
i)forX
iin thevalidation setVv.3. cross-validated MSEis
[
MSE
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
[Y
i^
k;v(X
i)]
2
:
We average the MSEs of theVvalidation sets. 15 / 48
Cross-validation
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
we obtain^
k;vusing thetraining set;we obtain^
k;v(X
i)forX
iin thevalidation setVv.3. cross-validated MSEis
[
MSE
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
[Y
i^
k;v(X
i)]
2
:
We average the MSEs of theVvalidation sets. 15 / 48
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
we obtain^
k;vusing thetraining set;we obtain^
k;v(X
i)forX
iin thevalidation setVv.3. cross-validated MSEis
[
MSE
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
[Y
i^
k;v(X
i)]
2
:
We average the MSEs of theVvalidation sets. 15 / 48
Cross-validation
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
we obtain^
k;vusing thetraining set;we obtain^
k;v(X
i)forX
iin thevalidation setVv.3. cross-validated MSEis
[
MSE
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
[Y
i^
k;v(X
i)]
2
:
We average the MSEs of theVvalidation sets. 15 / 48
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
we obtain^
k;vusing thetraining set;we obtain^
k;v(X
i)forX
iin thevalidation setVv.3. cross-validated MSEis
[
MSE
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
[Y
i^
k;v(X
i)]
2
:
We average the MSEs of theVvalidation sets. 15 / 48
Cross-validation
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
we obtain^
k;vusing thetraining set;we obtain^
k;v(X
i)forX
iin thevalidation setVv.3. cross-validated MSEis
[
MSE
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
[Y
i^
k;v(X
i)]
2
:
We average the MSEs of theVvalidation sets. 15 / 48
1. Vfolds of size roughlyn=V.
2. v=1; : : : ;V:
the data in folds other thanvis called thetraining set;
the data in foldvis called thetest/validation set.
we obtain^
k;vusing thetraining set;we obtain^
k;v(X
i)forX
iin thevalidation setVv.3. cross-validated MSEis
[
MSE
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
[Y
i^
k;v(X
i)]
2
:
We average the MSEs of theVvalidation sets. 15 / 48
123456789101Fold 1123456789102Fold 2123456789103Fold 3123456789104Fold 4123456789105Fold 5123456789106Fold 6123456789107Fold 7123456789108Fold 8123456789109Fold 91234567891010Fold 1012345678910CV preds.
Schematic of 10-fold cross-validation. Gray: training sets. Yellow: validation sets.
16 / 48
Schematic of 10-fold cross-validation. Gray: training sets. Yellow: validation sets.
16 / 48
How do we chooseV?
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
How do we chooseV?
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
How do we chooseV?
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
How do we chooseV?
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
How do we chooseV?
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
How do we chooseV?
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
How do we chooseV?
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
How do we chooseV?
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
How do we chooseV?
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
LargeV:
training data, so better forsmall n SmallV: test data
(People typically useV=5 orV=10.)
17 / 48
Discrete Super Learner
At this point, we have cross-validated MSE estimates
[
MSE
CV(^1); : : : ;
[
MSE
CV(^
K)
for each of our candidate algorithms.
We could simply take as our estimator the^
kminimizing
these cross-validated MSEs.
We call this the discrete Super Learner.
18 / 48
At this point, we have cross-validated MSE estimates
[
MSE
CV(^1); : : : ;
[
MSE
CV(^
K)
for each of our candidate algorithms.
We could simply take as our estimator the^
kminimizing
these cross-validated MSEs.
We call this the discrete Super Learner.
18 / 48
Discrete Super Learner
At this point, we have cross-validated MSE estimates
[
MSE
CV(^1); : : : ;
[
MSE
CV(^
K)
for each of our candidate algorithms.
We could simply take as our estimator the^
kminimizing
these cross-validated MSEs.
We call this the discrete Super Learner.
18 / 48
At this point, we have cross-validated MSE estimates
[
MSE
CV(^1); : : : ;
[
MSE
CV(^
K)
for each of our candidate algorithms.
We could simply take as our estimator the^
kminimizing
these cross-validated MSEs.
We call this the discrete Super Learner.
18 / 48
Discrete Super Learner
At this point, we have cross-validated MSE estimates
[
MSE
CV(^1); : : : ;
[
MSE
CV(^
K)
for each of our candidate algorithms.
We could simply take as our estimator the^
kminimizing
these cross-validated MSEs.
We call this the discrete Super Learner.
18 / 48
At this point, we have cross-validated MSE estimates
[
MSE
CV(^1); : : : ;
[
MSE
CV(^
K)
for each of our candidate algorithms.
We could simply take as our estimator the^
kminimizing
these cross-validated MSEs.
We call this the discrete Super Learner.
18 / 48
Super Learner
Let= (1; : : : ;
K)be an element ofS
K, the
K-dimensional simplex: each
k2[0;1]and
P
k
k=1.
Super Learner considers as its set of candidate algorithms
allconvex combinations^:=
P
K
k=1
k^
k:
The Super Learner is^
b
, where
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
(We use constrained optimization to compute the argmin.)
19 / 48
Let= (1; : : : ;
K)be an element ofS
K, the
K-dimensional simplex: each
k2[0;1]and
P
k
k=1.
Super Learner considers as its set of candidate algorithms
allconvex combinations^:=
P
K
k=1
k^
k:
The Super Learner is^
b
, where
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
(We use constrained optimization to compute the argmin.)
19 / 48
Super Learner
Let= (1; : : : ;
K)be an element ofS
K, the
K-dimensional simplex: each
k2[0;1]and
P
k
k=1.
Super Learner considers as its set of candidate algorithms
allconvex combinations^:=
P
K
k=1
k^
k:
The Super Learner is^
b
, where
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
(We use constrained optimization to compute the argmin.)
19 / 48
Let= (1; : : : ;
K)be an element ofS
K, the
K-dimensional simplex: each
k2[0;1]and
P
k
k=1.
Super Learner considers as its set of candidate algorithms
allconvex combinations^:=
P
K
k=1
k^
k:
The Super Learner is^
b
, where
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
(We use constrained optimization to compute the argmin.)
19 / 48
Super Learner
Let= (1; : : : ;
K)be an element ofS
K, the
K-dimensional simplex: each
k2[0;1]and
P
k
k=1.
Super Learner considers as its set of candidate algorithms
allconvex combinations^:=
P
K
k=1
k^
k:
The Super Learner is^
b
, where
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
(We use constrained optimization to compute the argmin.)
19 / 48
Let= (1; : : : ;
K)be an element ofS
K, the
K-dimensional simplex: each
k2[0;1]and
P
k
k=1.
Super Learner considers as its set of candidate algorithms
allconvex combinations^:=
P
K
k=1
k^
k:
The Super Learner is^
b
, where
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
(We use constrained optimization to compute the argmin.)
19 / 48
Super Learner
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
[
MSE
CV
K
X
k=1
k^
k
!
=
1
V
V
X
v=1
1
jVvj
X
i2Vv
"
Y
i
K
X
k=1
k^
k;v(X
i)
#2
:
20 / 48
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
[
MSE
CV
K
X
k=1
k^
k
!
=
1
V
V
X
v=1
1
jVvj
X
i2Vv
"
Y
i
K
X
k=1
k^
k;v(X
i)
#2
:
20 / 48
Super Learner
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
[
MSE
CV
K
X
k=1
k^
k
!
=
1
V
V
X
v=1
1
jVvj
X
i2Vv
"
Y
i
K
X
k=1
k^
k;v(X
i)
#2
:
20 / 48
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
[
MSE
CV
K
X
k=1
k^
k
!
=
1
V
V
X
v=1
1
jVvj
X
i2Vv
"
Y
i
K
X
k=1
k^
k;v(X
i)
#2
:
20 / 48
Super Learner
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
[
MSE
CV
K
X
k=1
k^
k
!
=
1
V
V
X
v=1
1
jVvj
X
i2Vv
"
Y
i
K
X
k=1
k^
k;v(X
i)
#2
:
20 / 48
b
:= arg min
2S
K
[
MSE
CV
K
X
k=1
k^
k
!
:
[
MSE
CV
K
X
k=1
k^
k
!
=
1
V
V
X
v=1
1
jVvj
X
i2Vv
"
Y
i
K
X
k=1
k^
k;v(X
i)
#2
:
20 / 48
Super Learner: steps
Putting it all together:
1. library of candidate algorithms^1; : : : ;^
K.2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
21 / 48
Putting it all together:
1. library of candidate algorithms^1; : : : ;^
K.2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
21 / 48
Super Learner: steps
Putting it all together:
1. library of candidate algorithms^1; : : : ;^
K.2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
21 / 48
Putting it all together:
1. library of candidate algorithms^1; : : : ;^
K.2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
21 / 48
Super Learner: steps
Putting it all together:
1. library of candidate algorithms^1; : : : ;^
K.2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
21 / 48
Putting it all together:
1. library of candidate algorithms^1; : : : ;^
K.2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
21 / 48
Super Learner: steps
Putting it all together:
1. library of candidate algorithms^1; : : : ;^
K.2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
21 / 48
Putting it all together:
1. library of candidate algorithms^1; : : : ;^
K.2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
21 / 48
Super Learner: steps
Putting it all together:
1. library of candidate algorithms^1; : : : ;^
K.2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
21 / 48
Putting it all together:
1. library of candidate algorithms^1; : : : ;^
K.2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
21 / 48
II.Lab1:
VanillaSLfora
continuousoutcome
21 / 48
VanillaSLfora
continuousoutcome
21 / 48
III.Intotheweeds:
amathematical
presentationofSL
21 / 48
amathematical
presentationofSL
21 / 48
Review
Recall the construction of SL for a continuous outcome:
1. library of candidate algorithms^1; : : : ;^
K.
2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.
3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
22 / 48
Recall the construction of SL for a continuous outcome:
1. library of candidate algorithms^1; : : : ;^
K.
2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.
3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
22 / 48
Review
Recall the construction of SL for a continuous outcome:
1. library of candidate algorithms^1; : : : ;^
K.
2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.
3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
22 / 48
Recall the construction of SL for a continuous outcome:
1. library of candidate algorithms^1; : : : ;^
K.
2. CV-predictions^
k;v(X
i)for allk;vandi2 Vv.
3. SL weights
b
:= arg min
2S
K
[
MSE
CV
P
K
k=1
k^
k
:
4. ^
SL=
P
K
k=1
b
k^
k.
22 / 48
In this section, we generalize this procedure to estimation
of
appropriate loss
23 / 48
of
appropriate loss
23 / 48
Loss and risk: setup
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Loss and risk: setup
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Loss and risk: setup
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Loss and risk: setup
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Loss and risk: setup
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Loss and risk: setup
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Loss and risk: setup
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Denote byOtheobserved data unit e.g.O= (Y;X).
Denote byOthesample spaceofOLetMdenote ourstatistical model.Denote byP02 Mthetrue distributionofO.Thus, we observe i.i.d. copiesO1; : : : ;OnP0.Suppose we want to estimate aparameter:M !.Denote0:=(P0)the true parameter value.
24 / 48
Loss and risk
LetLbe a map fromO toR.
We callLaloss functionforif it holds that
0= arg min
2
E
P0
[L(O; )]:
R0() =E
P0
[L(O; )]is called theoracle risk.
These denitions of loss and risk come from thestatistical
learningliterature (see, e.g. Vapnik, 1992, 1999, 2013)
and arenot to be confusedwith loss and risk from the
decision theory literature (e.g. Ferguson, 2014).
25 / 48
LetLbe a map fromO toR.
We callLaloss functionforif it holds that
0= arg min
2
E
P0
[L(O; )]:
R0() =E
P0
[L(O; )]is called theoracle risk.
These denitions of loss and risk come from thestatistical
learningliterature (see, e.g. Vapnik, 1992, 1999, 2013)
and arenot to be confusedwith loss and risk from the
decision theory literature (e.g. Ferguson, 2014).
25 / 48
Loss and risk
LetLbe a map fromO toR.
We callLaloss functionforif it holds that
0= arg min
2
E
P0
[L(O; )]:
R0() =E
P0
[L(O; )]is called theoracle risk.
These denitions of loss and risk come from thestatistical
learningliterature (see, e.g. Vapnik, 1992, 1999, 2013)
and arenot to be confusedwith loss and risk from the
decision theory literature (e.g. Ferguson, 2014).
25 / 48
LetLbe a map fromO toR.
We callLaloss functionforif it holds that
0= arg min
2
E
P0
[L(O; )]:
R0() =E
P0
[L(O; )]is called theoracle risk.
These denitions of loss and risk come from thestatistical
learningliterature (see, e.g. Vapnik, 1992, 1999, 2013)
and arenot to be confusedwith loss and risk from the
decision theory literature (e.g. Ferguson, 2014).
25 / 48
Loss and risk
LetLbe a map fromO toR.
We callLaloss functionforif it holds that
0= arg min
2
E
P0
[L(O; )]:
R0() =E
P0
[L(O; )]is called theoracle risk.
These denitions of loss and risk come from thestatistical
learningliterature (see, e.g. Vapnik, 1992, 1999, 2013)
and arenot to be confusedwith loss and risk from the
decision theory literature (e.g. Ferguson, 2014).
25 / 48
LetLbe a map fromO toR.
We callLaloss functionforif it holds that
0= arg min
2
E
P0
[L(O; )]:
R0() =E
P0
[L(O; )]is called theoracle risk.
These denitions of loss and risk come from thestatistical
learningliterature (see, e.g. Vapnik, 1992, 1999, 2013)
and arenot to be confusedwith loss and risk from the
decision theory literature (e.g. Ferguson, 2014).
25 / 48
Loss and risk
LetLbe a map fromO toR.
We callLaloss functionforif it holds that
0= arg min
2
E
P0
[L(O; )]:
R0() =E
P0
[L(O; )]is called theoracle risk.
These denitions of loss and risk come from thestatistical
learningliterature (see, e.g. Vapnik, 1992, 1999, 2013)
and arenot to be confusedwith loss and risk from the
decision theory literature (e.g. Ferguson, 2014).
25 / 48
LetLbe a map fromO toR.
We callLaloss functionforif it holds that
0= arg min
2
E
P0
[L(O; )]:
R0() =E
P0
[L(O; )]is called theoracle risk.
These denitions of loss and risk come from thestatistical
learningliterature (see, e.g. Vapnik, 1992, 1999, 2013)
and arenot to be confusedwith loss and risk from the
decision theory literature (e.g. Ferguson, 2014).
25 / 48
Loss and risk: MSE example
MSE is the oracle risk corresponding to a
squared-error loss function
O= (Y;X).
(P) =(P) =fx7!E
P[YjX=x]gL(O; ) = [Y(X)]
2
is thesquared-error loss.R0() =MSE() =E
P0
[Y(X)]
2
.
26 / 48
MSE is the oracle risk corresponding to a
squared-error loss function
O= (Y;X).
(P) =(P) =fx7!E
P[YjX=x]gL(O; ) = [Y(X)]
2
is thesquared-error loss.R0() =MSE() =E
P0
[Y(X)]
2
.
26 / 48
Loss and risk: MSE example
MSE is the oracle risk corresponding to a
squared-error loss function
O= (Y;X).
(P) =(P) =fx7!E
P[YjX=x]gL(O; ) = [Y(X)]
2
is thesquared-error loss.R0() =MSE() =E
P0
[Y(X)]
2
.
26 / 48
MSE is the oracle risk corresponding to a
squared-error loss function
O= (Y;X).
(P) =(P) =fx7!E
P[YjX=x]gL(O; ) = [Y(X)]
2
is thesquared-error loss.R0() =MSE() =E
P0
[Y(X)]
2
.
26 / 48
Loss and risk: MSE example
MSE is the oracle risk corresponding to a
squared-error loss function
O= (Y;X).
(P) =(P) =fx7!E
P[YjX=x]gL(O; ) = [Y(X)]
2
is thesquared-error loss.R0() =MSE() =E
P0
[Y(X)]
2
.
26 / 48
MSE is the oracle risk corresponding to a
squared-error loss function
O= (Y;X).
(P) =(P) =fx7!E
P[YjX=x]gL(O; ) = [Y(X)]
2
is thesquared-error loss.R0() =MSE() =E
P0
[Y(X)]
2
.
26 / 48
Loss and risk: MSE example
MSE is the oracle risk corresponding to a
squared-error loss function
O= (Y;X).
(P) =(P) =fx7!E
P[YjX=x]gL(O; ) = [Y(X)]
2
is thesquared-error loss.R0() =MSE() =E
P0
[Y(X)]
2
.
26 / 48
MSE is the oracle risk corresponding to a
squared-error loss function
O= (Y;X).
(P) =(P) =fx7!E
P[YjX=x]gL(O; ) = [Y(X)]
2
is thesquared-error loss.R0() =MSE() =E
P0
[Y(X)]
2
.
26 / 48
Loss and risk: MSE example
MSE is the oracle risk corresponding to a
squared-error loss function
O= (Y;X).
(P) =(P) =fx7!E
P[YjX=x]gL(O; ) = [Y(X)]
2
is thesquared-error loss.R0() =MSE() =E
P0
[Y(X)]
2
.
26 / 48
MSE is the oracle risk corresponding to a
squared-error loss function
O= (Y;X).
(P) =(P) =fx7!E
P[YjX=x]gL(O; ) = [Y(X)]
2
is thesquared-error loss.R0() =MSE() =E
P0
[Y(X)]
2
.
26 / 48
Estimating the oracle risk
0= arg min
2
R0()
R0() =E
P0
[L(O; )]
Suppose that
^
1; : : : ;
^
Kare candidate estimators.
As before, we need to estimateR0()to evaluate each^
k.The naive estimator is
b
R(
^
k) =
1
n
P
n
i=1
L(O
i;
^
k).We instead estimateR0()using thecross-validated risk
b
R
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
L(O
i;^
k;v):
27 / 48
0= arg min
2
R0()
R0() =E
P0
[L(O; )]
Suppose that
^
1; : : : ;
^
Kare candidate estimators.
As before, we need to estimateR0()to evaluate each^
k.The naive estimator is
b
R(
^
k) =
1
n
P
n
i=1
L(O
i;
^
k).We instead estimateR0()using thecross-validated risk
b
R
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
L(O
i;^
k;v):
27 / 48
Estimating the oracle risk
0= arg min
2
R0()
R0() =E
P0
[L(O; )]
Suppose that
^
1; : : : ;
^
Kare candidate estimators.
As before, we need to estimateR0()to evaluate each^
k.The naive estimator is
b
R(
^
k) =
1
n
P
n
i=1
L(O
i;
^
k).We instead estimateR0()using thecross-validated risk
b
R
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
L(O
i;^
k;v):
27 / 48
0= arg min
2
R0()
R0() =E
P0
[L(O; )]
Suppose that
^
1; : : : ;
^
Kare candidate estimators.
As before, we need to estimateR0()to evaluate each^
k.The naive estimator is
b
R(
^
k) =
1
n
P
n
i=1
L(O
i;
^
k).We instead estimateR0()using thecross-validated risk
b
R
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
L(O
i;^
k;v):
27 / 48
Estimating the oracle risk
0= arg min
2
R0()
R0() =E
P0
[L(O; )]
Suppose that
^
1; : : : ;
^
Kare candidate estimators.
As before, we need to estimateR0()to evaluate each^
k.The naive estimator is
b
R(
^
k) =
1
n
P
n
i=1
L(O
i;
^
k).We instead estimateR0()using thecross-validated risk
b
R
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
L(O
i;^
k;v):
27 / 48
0= arg min
2
R0()
R0() =E
P0
[L(O; )]
Suppose that
^
1; : : : ;
^
Kare candidate estimators.
As before, we need to estimateR0()to evaluate each^
k.The naive estimator is
b
R(
^
k) =
1
n
P
n
i=1
L(O
i;
^
k).We instead estimateR0()using thecross-validated risk
b
R
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
L(O
i;^
k;v):
27 / 48
Estimating the oracle risk
0= arg min
2
R0()
R0() =E
P0
[L(O; )]
Suppose that
^
1; : : : ;
^
Kare candidate estimators.
As before, we need to estimateR0()to evaluate each^
k.The naive estimator is
b
R(
^
k) =
1
n
P
n
i=1
L(O
i;
^
k).We instead estimateR0()using thecross-validated risk
b
R
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
L(O
i;^
k;v):
27 / 48
0= arg min
2
R0()
R0() =E
P0
[L(O; )]
Suppose that
^
1; : : : ;
^
Kare candidate estimators.
As before, we need to estimateR0()to evaluate each^
k.The naive estimator is
b
R(
^
k) =
1
n
P
n
i=1
L(O
i;
^
k).We instead estimateR0()using thecross-validated risk
b
R
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
L(O
i;^
k;v):
27 / 48
Estimating the oracle risk
0= arg min
2
R0()
R0() =E
P0
[L(O; )]
Suppose that
^
1; : : : ;
^
Kare candidate estimators.
As before, we need to estimateR0()to evaluate each^
k.The naive estimator is
b
R(
^
k) =
1
n
P
n
i=1
L(O
i;
^
k).We instead estimateR0()using thecross-validated risk
b
R
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
L(O
i;^
k;v):
27 / 48
0= arg min
2
R0()
R0() =E
P0
[L(O; )]
Suppose that
^
1; : : : ;
^
Kare candidate estimators.
As before, we need to estimateR0()to evaluate each^
k.The naive estimator is
b
R(
^
k) =
1
n
P
n
i=1
L(O
i;
^
k).We instead estimateR0()using thecross-validated risk
b
R
CV(^
k) =
1
V
V
X
v=1
1
jVvj
X
i2Vv
L(O
i;^
k;v):
27 / 48
Super Learner: general steps
Using this framework, we can generalize the SL recipe:
1. library of candidate algorithms
^
1; : : : ;
^
K.2. CV-Risks
b
R
CV(^
k),k=1; : : : ;K.3. SL weights
b
:= arg min
2S
K
b
R
CV
P
K
k=1
k
^
k
:
4. ^
SL=
P
K
k=1
b
k
^
k.
28 / 48
Using this framework, we can generalize the SL recipe:
1. library of candidate algorithms
^
1; : : : ;
^
K.2. CV-Risks
b
R
CV(^
k),k=1; : : : ;K.3. SL weights
b
:= arg min
2S
K
b
R
CV
P
K
k=1
k
^
k
:
4. ^
SL=
P
K
k=1
b
k
^
k.
28 / 48
Super Learner: general steps
Using this framework, we can generalize the SL recipe:
1. library of candidate algorithms
^
1; : : : ;
^
K.2. CV-Risks
b
R
CV(^
k),k=1; : : : ;K.3. SL weights
b
:= arg min
2S
K
b
R
CV
P
K
k=1
k
^
k
:
4. ^
SL=
P
K
k=1
b
k
^
k.
28 / 48
Using this framework, we can generalize the SL recipe:
1. library of candidate algorithms
^
1; : : : ;
^
K.2. CV-Risks
b
R
CV(^
k),k=1; : : : ;K.3. SL weights
b
:= arg min
2S
K
b
R
CV
P
K
k=1
k
^
k
:
4. ^
SL=
P
K
k=1
b
k
^
k.
28 / 48
Super Learner: general steps
Using this framework, we can generalize the SL recipe:
1. library of candidate algorithms
^
1; : : : ;
^
K.2. CV-Risks
b
R
CV(^
k),k=1; : : : ;K.3. SL weights
b
:= arg min
2S
K
b
R
CV
P
K
k=1
k
^
k
:
4. ^
SL=
P
K
k=1
b
k
^
k.
28 / 48
Using this framework, we can generalize the SL recipe:
1. library of candidate algorithms
^
1; : : : ;
^
K.2. CV-Risks
b
R
CV(^
k),k=1; : : : ;K.3. SL weights
b
:= arg min
2S
K
b
R
CV
P
K
k=1
k
^
k
:
4. ^
SL=
P
K
k=1
b
k
^
k.
28 / 48
Super Learner: general steps
Using this framework, we can generalize the SL recipe:
1. library of candidate algorithms
^
1; : : : ;
^
K.2. CV-Risks
b
R
CV(^
k),k=1; : : : ;K.3. SL weights
b
:= arg min
2S
K
b
R
CV
P
K
k=1
k
^
k
:
4. ^
SL=
P
K
k=1
b
k
^
k.
28 / 48
Using this framework, we can generalize the SL recipe:
1. library of candidate algorithms
^
1; : : : ;
^
K.2. CV-Risks
b
R
CV(^
k),k=1; : : : ;K.3. SL weights
b
:= arg min
2S
K
b
R
CV
P
K
k=1
k
^
k
:
4. ^
SL=
P
K
k=1
b
k
^
k.
28 / 48
Super Learner: general steps
Using this framework, we can generalize the SL recipe:
1. library of candidate algorithms
^
1; : : : ;
^
K.2. CV-Risks
b
R
CV(^
k),k=1; : : : ;K.3. SL weights
b
:= arg min
2S
K
b
R
CV
P
K
k=1
k
^
k
:
4. ^
SL=
P
K
k=1
b
k
^
k.
28 / 48
Using this framework, we can generalize the SL recipe:
1. library of candidate algorithms
^
1; : : : ;
^
K.2. CV-Risks
b
R
CV(^
k),k=1; : : : ;K.3. SL weights
b
:= arg min
2S
K
b
R
CV
P
K
k=1
k
^
k
:
4. ^
SL=
P
K
k=1
b
k
^
k.
28 / 48
Theoretical guarantees
van der Vaart et al. (2006) showed that, under some conditions,
theoracle risk of the SL estimatorisas goodas theoracle
risk of the oracle minimizerup to a multiple of
logn
n
as long as
the number of candidate algorithms ispolynomial inn.
29 / 48
van der Vaart et al. (2006) showed that, under some conditions,
theoracle risk of the SL estimatorisas goodas theoracle
risk of the oracle minimizerup to a multiple of
logn
n
as long as
the number of candidate algorithms ispolynomial inn.
29 / 48
Loss functions for a binary
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
Loss functions for a binary
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
Loss functions for a binary
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
Loss functions for a binary
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
Loss functions for a binary
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
Loss functions for a binary
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
outcome
We return toO= (Y;X),=.
ForcontinuousY, we usedsquared-error loss.
ForbinaryY, squared-error loss is still valid.However, there are (at least) two other alternative loss
functions for a binary outcome.
Negative log-likelihood loss:
L(O; ) =Ylog(X)[1Y] log[1(X)].
AUC loss.
30 / 48
IV.Lab2:
VanillaSLforabinary
outcome
30 / 48
VanillaSLforabinary
outcome
30 / 48
15minutebreak
30 / 48
30 / 48
V.Bellsandwhistles:
Screens,weights,and
CV-SL
30 / 48
Screens,weights,and
CV-SL
30 / 48
Overview
In this section, we will introduce three of the add-ons to SL that
are frequently useful in practice:variable screens,
observation weights, andcross-validated SL.
31 / 48
In this section, we will introduce three of the add-ons to SL that
are frequently useful in practice:variable screens,
observation weights, andcross-validated SL.
31 / 48
Variable screens
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
Variable screens
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
Variable screens
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
Variable screens
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
Variable screens
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
Variable screens
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
We think of a candidate algorithm as a two-step procedure:
1.Select a subsetof the covariates.2. t a model.We call step 1 ascreening procedure.While we could program steps 1 and 2 by hand in to each
candidate algorithm, theSuperLearnerpackage has
built-in functionality to ease this process.
Screening algorithms allow us to
domain knowledge.
32 / 48
Example use-cases of screening
If we have a high-dimensional set of covariates, we can try
different ways ofreducing the dimensionality.
If we have a large number of raw measurements, we
might try providing a smaller number ofsummary
measures e.g. mean, median, min, max.
If we have measurements collected atmultiple time
points, we might try providing just baseline, or just the last
time point, or some summaries of the trajectory.
We can force certain variables to always be used.
33 / 48
If we have a high-dimensional set of covariates, we can try
different ways ofreducing the dimensionality.
If we have a large number of raw measurements, we
might try providing a smaller number ofsummary
measures e.g. mean, median, min, max.
If we have measurements collected atmultiple time
points, we might try providing just baseline, or just the last
time point, or some summaries of the trajectory.
We can force certain variables to always be used.
33 / 48
Example use-cases of screening
If we have a high-dimensional set of covariates, we can try
different ways ofreducing the dimensionality.
If we have a large number of raw measurements, we
might try providing a smaller number ofsummary
measures e.g. mean, median, min, max.
If we have measurements collected atmultiple time
points, we might try providing just baseline, or just the last
time point, or some summaries of the trajectory.
We can force certain variables to always be used.
33 / 48
If we have a high-dimensional set of covariates, we can try
different ways ofreducing the dimensionality.
If we have a large number of raw measurements, we
might try providing a smaller number ofsummary
measures e.g. mean, median, min, max.
If we have measurements collected atmultiple time
points, we might try providing just baseline, or just the last
time point, or some summaries of the trajectory.
We can force certain variables to always be used.
33 / 48
Example use-cases of screening
If we have a high-dimensional set of covariates, we can try
different ways ofreducing the dimensionality.
If we have a large number of raw measurements, we
might try providing a smaller number ofsummary
measures e.g. mean, median, min, max.
If we have measurements collected atmultiple time
points, we might try providing just baseline, or just the last
time point, or some summaries of the trajectory.
We can force certain variables to always be used.
33 / 48
If we have a high-dimensional set of covariates, we can try
different ways ofreducing the dimensionality.
If we have a large number of raw measurements, we
might try providing a smaller number ofsummary
measures e.g. mean, median, min, max.
If we have measurements collected atmultiple time
points, we might try providing just baseline, or just the last
time point, or some summaries of the trajectory.
We can force certain variables to always be used.
33 / 48
Example use-cases of screening
If we have a high-dimensional set of covariates, we can try
different ways ofreducing the dimensionality.
If we have a large number of raw measurements, we
might try providing a smaller number ofsummary
measures e.g. mean, median, min, max.
If we have measurements collected atmultiple time
points, we might try providing just baseline, or just the last
time point, or some summaries of the trajectory.
We can force certain variables to always be used.
33 / 48
If we have a high-dimensional set of covariates, we can try
different ways ofreducing the dimensionality.
If we have a large number of raw measurements, we
might try providing a smaller number ofsummary
measures e.g. mean, median, min, max.
If we have measurements collected atmultiple time
points, we might try providing just baseline, or just the last
time point, or some summaries of the trajectory.
We can force certain variables to always be used.
33 / 48
Observation weights
In some applications, we need to includeobservation
weightsin the procedure e.g.case-control sampling,
or as a simple way to account forloss-to-followup.
Observation weights can be included directly in a call to
SuperLearner, butmethod.AUC does not make correct
use of weights!!!!
Note that someSuperLearnerwrappers might not make
use of observation weights.
34 / 48
In some applications, we need to includeobservation
weightsin the procedure e.g.case-control sampling,
or as a simple way to account forloss-to-followup.
Observation weights can be included directly in a call to
SuperLearner, butmethod.AUC does not make correct
use of weights!!!!
Note that someSuperLearnerwrappers might not make
use of observation weights.
34 / 48
Observation weights
In some applications, we need to includeobservation
weightsin the procedure e.g.case-control sampling,
or as a simple way to account forloss-to-followup.
Observation weights can be included directly in a call to
SuperLearner, butmethod.AUC does not make correct
use of weights!!!!
Note that someSuperLearnerwrappers might not make
use of observation weights.
34 / 48
In some applications, we need to includeobservation
weightsin the procedure e.g.case-control sampling,
or as a simple way to account forloss-to-followup.
Observation weights can be included directly in a call to
SuperLearner, butmethod.AUC does not make correct
use of weights!!!!
Note that someSuperLearnerwrappers might not make
use of observation weights.
34 / 48
Observation weights
In some applications, we need to includeobservation
weightsin the procedure e.g.case-control sampling,
or as a simple way to account forloss-to-followup.
Observation weights can be included directly in a call to
SuperLearner, butmethod.AUC does not make correct
use of weights!!!!
Note that someSuperLearnerwrappers might not make
use of observation weights.
34 / 48
In some applications, we need to includeobservation
weightsin the procedure e.g.case-control sampling,
or as a simple way to account forloss-to-followup.
Observation weights can be included directly in a call to
SuperLearner, butmethod.AUC does not make correct
use of weights!!!!
Note that someSuperLearnerwrappers might not make
use of observation weights.
34 / 48
Case-control weights
LetYrepresent disease status at the end of a study.
Suppose specimens from allncasecases(Y
i=1) are
assayed.
A random subset ofN
controlcontrols(Y
i=0) (out of
n
controltotal controls) are assayed.
We will use this case-control cohort to predict disease
status using the results of the assay and other covariates.
35 / 48
LetYrepresent disease status at the end of a study.
Suppose specimens from allncasecases(Y
i=1) are
assayed.
A random subset ofN
controlcontrols(Y
i=0) (out of
n
controltotal controls) are assayed.
We will use this case-control cohort to predict disease
status using the results of the assay and other covariates.
35 / 48
Case-control weights
LetYrepresent disease status at the end of a study.
Suppose specimens from allncasecases(Y
i=1) are
assayed.
A random subset ofN
controlcontrols(Y
i=0) (out of
n
controltotal controls) are assayed.
We will use this case-control cohort to predict disease
status using the results of the assay and other covariates.
35 / 48
LetYrepresent disease status at the end of a study.
Suppose specimens from allncasecases(Y
i=1) are
assayed.
A random subset ofN
controlcontrols(Y
i=0) (out of
n
controltotal controls) are assayed.
We will use this case-control cohort to predict disease
status using the results of the assay and other covariates.
35 / 48
Case-control weights
LetYrepresent disease status at the end of a study.
Suppose specimens from allncasecases(Y
i=1) are
assayed.
A random subset ofN
controlcontrols(Y
i=0) (out of
n
controltotal controls) are assayed.
We will use this case-control cohort to predict disease
status using the results of the assay and other covariates.
35 / 48
LetYrepresent disease status at the end of a study.
Suppose specimens from allncasecases(Y
i=1) are
assayed.
A random subset ofN
controlcontrols(Y
i=0) (out of
n
controltotal controls) are assayed.
We will use this case-control cohort to predict disease
status using the results of the assay and other covariates.
35 / 48
Case-control weights
LetYrepresent disease status at the end of a study.
Suppose specimens from allncasecases(Y
i=1) are
assayed.
A random subset ofN
controlcontrols(Y
i=0) (out of
n
controltotal controls) are assayed.
We will use this case-control cohort to predict disease
status using the results of the assay and other covariates.
35 / 48
LetYrepresent disease status at the end of a study.
Suppose specimens from allncasecases(Y
i=1) are
assayed.
A random subset ofN
controlcontrols(Y
i=0) (out of
n
controltotal controls) are assayed.
We will use this case-control cohort to predict disease
status using the results of the assay and other covariates.
35 / 48
Case-control weights
We can use SL with observation weights.
Caseshave weightw
i=1.Controlshave weightw
i=n
control=N
control.Control weights could also be estimated using a logistic
regression of the indicator of inclusion in the control cohort
on baseline covariates.
36 / 48
We can use SL with observation weights.
Caseshave weightw
i=1.Controlshave weightw
i=n
control=N
control.Control weights could also be estimated using a logistic
regression of the indicator of inclusion in the control cohort
on baseline covariates.
36 / 48
Case-control weights
We can use SL with observation weights.
Caseshave weightw
i=1.Controlshave weightw
i=n
control=N
control.Control weights could also be estimated using a logistic
regression of the indicator of inclusion in the control cohort
on baseline covariates.
36 / 48
We can use SL with observation weights.
Caseshave weightw
i=1.Controlshave weightw
i=n
control=N
control.Control weights could also be estimated using a logistic
regression of the indicator of inclusion in the control cohort
on baseline covariates.
36 / 48
Case-control weights
We can use SL with observation weights.
Caseshave weightw
i=1.Controlshave weightw
i=n
control=N
control.Control weights could also be estimated using a logistic
regression of the indicator of inclusion in the control cohort
on baseline covariates.
36 / 48
We can use SL with observation weights.
Caseshave weightw
i=1.Controlshave weightw
i=n
control=N
control.Control weights could also be estimated using a logistic
regression of the indicator of inclusion in the control cohort
on baseline covariates.
36 / 48
Case-control weights
We can use SL with observation weights.
Caseshave weightw
i=1.Controlshave weightw
i=n
control=N
control.Control weights could also be estimated using a logistic
regression of the indicator of inclusion in the control cohort
on baseline covariates.
36 / 48
We can use SL with observation weights.
Caseshave weightw
i=1.Controlshave weightw
i=n
control=N
control.Control weights could also be estimated using a logistic
regression of the indicator of inclusion in the control cohort
on baseline covariates.
36 / 48
Right-censored outcomes
SupposeY=I(Tt0)indicates that disease occurs
before timet0.
Tis subject to right-censoring byC: we observe
Y= minfT;Cgand =I(TC).
We want to estimate
(x) =P(Tt0jX=x) =E[YjX=x].
37 / 48
SupposeY=I(Tt0)indicates that disease occurs
before timet0.
Tis subject to right-censoring byC: we observe
Y= minfT;Cgand =I(TC).
We want to estimate
(x) =P(Tt0jX=x) =E[YjX=x].
37 / 48
Right-censored outcomes
SupposeY=I(Tt0)indicates that disease occurs
before timet0.
Tis subject to right-censoring byC: we observe
Y= minfT;Cgand =I(TC).
We want to estimate
(x) =P(Tt0jX=x) =E[YjX=x].
37 / 48
SupposeY=I(Tt0)indicates that disease occurs
before timet0.
Tis subject to right-censoring byC: we observe
Y= minfT;Cgand =I(TC).
We want to estimate
(x) =P(Tt0jX=x) =E[YjX=x].
37 / 48
Right-censored outcomes
SupposeY=I(Tt0)indicates that disease occurs
before timet0.
Tis subject to right-censoring byC: we observe
Y= minfT;Cgand =I(TC).
We want to estimate
(x) =P(Tt0jX=x) =E[YjX=x].
37 / 48
SupposeY=I(Tt0)indicates that disease occurs
before timet0.
Tis subject to right-censoring byC: we observe
Y= minfT;Cgand =I(TC).
We want to estimate
(x) =P(Tt0jX=x) =E[YjX=x].
37 / 48
Right-censored outcomes
0= arg min
E
P0
G0(YjX)
L((Y;X); )
Here,G0(tjx) =P0(C>tjX=x).
Leither squared-error or negative log-likelihood loss.
If we knewG0, we could use SL with weight
G0(YjX)
.Instead, we estimateG0and plug in this estimator to
obtain an estimated weight.
IfC??T, we can use a Kaplan-Meier estimator forG0;
otherwise we might use a Cox model.
38 / 48
0= arg min
E
P0
G0(YjX)
L((Y;X); )
Here,G0(tjx) =P0(C>tjX=x).
Leither squared-error or negative log-likelihood loss.
If we knewG0, we could use SL with weight
G0(YjX)
.Instead, we estimateG0and plug in this estimator to
obtain an estimated weight.
IfC??T, we can use a Kaplan-Meier estimator forG0;
otherwise we might use a Cox model.
38 / 48
Right-censored outcomes
0= arg min
E
P0
G0(YjX)
L((Y;X); )
Here,G0(tjx) =P0(C>tjX=x).
Leither squared-error or negative log-likelihood loss.
If we knewG0, we could use SL with weight
G0(YjX)
.Instead, we estimateG0and plug in this estimator to
obtain an estimated weight.
IfC??T, we can use a Kaplan-Meier estimator forG0;
otherwise we might use a Cox model.
38 / 48
0= arg min
E
P0
G0(YjX)
L((Y;X); )
Here,G0(tjx) =P0(C>tjX=x).
Leither squared-error or negative log-likelihood loss.
If we knewG0, we could use SL with weight
G0(YjX)
.Instead, we estimateG0and plug in this estimator to
obtain an estimated weight.
IfC??T, we can use a Kaplan-Meier estimator forG0;
otherwise we might use a Cox model.
38 / 48
Right-censored outcomes
0= arg min
E
P0
G0(YjX)
L((Y;X); )
Here,G0(tjx) =P0(C>tjX=x).
Leither squared-error or negative log-likelihood loss.
If we knewG0, we could use SL with weight
G0(YjX)
.Instead, we estimateG0and plug in this estimator to
obtain an estimated weight.
IfC??T, we can use a Kaplan-Meier estimator forG0;
otherwise we might use a Cox model.
38 / 48
0= arg min
E
P0
G0(YjX)
L((Y;X); )
Here,G0(tjx) =P0(C>tjX=x).
Leither squared-error or negative log-likelihood loss.
If we knewG0, we could use SL with weight
G0(YjX)
.Instead, we estimateG0and plug in this estimator to
obtain an estimated weight.
IfC??T, we can use a Kaplan-Meier estimator forG0;
otherwise we might use a Cox model.
38 / 48
Right-censored outcomes
0= arg min
E
P0
G0(YjX)
L((Y;X); )
Here,G0(tjx) =P0(C>tjX=x).
Leither squared-error or negative log-likelihood loss.
If we knewG0, we could use SL with weight
G0(YjX)
.Instead, we estimateG0and plug in this estimator to
obtain an estimated weight.
IfC??T, we can use a Kaplan-Meier estimator forG0;
otherwise we might use a Cox model.
38 / 48
0= arg min
E
P0
G0(YjX)
L((Y;X); )
Here,G0(tjx) =P0(C>tjX=x).
Leither squared-error or negative log-likelihood loss.
If we knewG0, we could use SL with weight
G0(YjX)
.Instead, we estimateG0and plug in this estimator to
obtain an estimated weight.
IfC??T, we can use a Kaplan-Meier estimator forG0;
otherwise we might use a Cox model.
38 / 48
CV-Super Learner
The standard SL framework gives us CV risks for each
candidate algorithm.
However, the SL and discrete SL are obtained using all the
data, sotheir estimated risks will be optimistic.
We can rectify this using asecond layer of
cross-validation.
39 / 48
The standard SL framework gives us CV risks for each
candidate algorithm.
However, the SL and discrete SL are obtained using all the
data, sotheir estimated risks will be optimistic.
We can rectify this using asecond layer of
cross-validation.
39 / 48
CV-Super Learner
The standard SL framework gives us CV risks for each
candidate algorithm.
However, the SL and discrete SL are obtained using all the
data, sotheir estimated risks will be optimistic.
We can rectify this using asecond layer of
cross-validation.
39 / 48
The standard SL framework gives us CV risks for each
candidate algorithm.
However, the SL and discrete SL are obtained using all the
data, sotheir estimated risks will be optimistic.
We can rectify this using asecond layer of
cross-validation.
39 / 48
CV-Super Learner
The standard SL framework gives us CV risks for each
candidate algorithm.
However, the SL and discrete SL are obtained using all the
data, sotheir estimated risks will be optimistic.
We can rectify this using asecond layer of
cross-validation.
39 / 48
The standard SL framework gives us CV risks for each
candidate algorithm.
However, the SL and discrete SL are obtained using all the
data, sotheir estimated risks will be optimistic.
We can rectify this using asecond layer of
cross-validation.
39 / 48
CV-Super Learner
The standard SL framework gives us CV risks for each
candidate algorithm.
However, the SL and discrete SL are obtained using all the
data, sotheir estimated risks will be optimistic.
We can rectify this using asecond layer of
cross-validation.
39 / 48
The standard SL framework gives us CV risks for each
candidate algorithm.
However, the SL and discrete SL are obtained using all the
data, sotheir estimated risks will be optimistic.
We can rectify this using asecond layer of
cross-validation.
39 / 48
CV-Super Learner
1. V1folds.
2. v=1; : : : ;V1:a. vusing
V2-fold CV.
b.
validation set for foldv.
3.
discrete SL and SL.
40 / 48
1. V1folds.
2. v=1; : : : ;V1:a. vusing
V2-fold CV.
b.
validation set for foldv.
3.
discrete SL and SL.
40 / 48
CV-Super Learner
1. V1folds.
2. v=1; : : : ;V1:a. vusing
V2-fold CV.
b.
validation set for foldv.
3.
discrete SL and SL.
40 / 48
1. V1folds.
2. v=1; : : : ;V1:a. vusing
V2-fold CV.
b.
validation set for foldv.
3.
discrete SL and SL.
40 / 48
CV-Super Learner
1. V1folds.
2. v=1; : : : ;V1:a. vusing
V2-fold CV.
b.
validation set for foldv.
3.
discrete SL and SL.
40 / 48
1. V1folds.
2. v=1; : : : ;V1:a. vusing
V2-fold CV.
b.
validation set for foldv.
3.
discrete SL and SL.
40 / 48
CV-Super Learner
1. V1folds.
2. v=1; : : : ;V1:a. vusing
V2-fold CV.
b.
validation set for foldv.
3.
discrete SL and SL.
40 / 48
1. V1folds.
2. v=1; : : : ;V1:a. vusing
V2-fold CV.
b.
validation set for foldv.
3.
discrete SL and SL.
40 / 48
CV-Super Learner
1. V1folds.
2. v=1; : : : ;V1:a. vusing
V2-fold CV.
b.
validation set for foldv.
3.
discrete SL and SL.
40 / 48
1. V1folds.
2. v=1; : : : ;V1:a. vusing
V2-fold CV.
b.
validation set for foldv.
3.
discrete SL and SL.
40 / 48
VI.Lab3:
Binaryoutcome
redux
40 / 48
Binaryoutcome
redux
40 / 48
VII.Lab4:
Case-controlanalysis
ofFluzonevaccine
40 / 48
Case-controlanalysis
ofFluzonevaccine
40 / 48
FLUVACS trial
Health adults aged 1849 years, Michigan, 20072008.
Randomly assigned to:
We are only interested in Fluzone vs placebo.Followed for one u season.
Endpoint = laboratory-conrmed inuenza.
41 / 48
Health adults aged 1849 years, Michigan, 20072008.
Randomly assigned to:
We are only interested in Fluzone vs placebo.Followed for one u season.
Endpoint = laboratory-conrmed inuenza.
41 / 48
FLUVACS trial
Health adults aged 1849 years, Michigan, 20072008.
Randomly assigned to:
We are only interested in Fluzone vs placebo.Followed for one u season.
Endpoint = laboratory-conrmed inuenza.
41 / 48
Health adults aged 1849 years, Michigan, 20072008.
Randomly assigned to:
We are only interested in Fluzone vs placebo.Followed for one u season.
Endpoint = laboratory-conrmed inuenza.
41 / 48
FLUVACS trial
Health adults aged 1849 years, Michigan, 20072008.
Randomly assigned to:
We are only interested in Fluzone vs placebo.Followed for one u season.
Endpoint = laboratory-conrmed inuenza.
41 / 48
Health adults aged 1849 years, Michigan, 20072008.
Randomly assigned to:
We are only interested in Fluzone vs placebo.Followed for one u season.
Endpoint = laboratory-conrmed inuenza.
41 / 48
FLUVACS trial
Health adults aged 1849 years, Michigan, 20072008.
Randomly assigned to:
We are only interested in Fluzone vs placebo.Followed for one u season.
Endpoint = laboratory-conrmed inuenza.
41 / 48
Health adults aged 1849 years, Michigan, 20072008.
Randomly assigned to:
We are only interested in Fluzone vs placebo.Followed for one u season.
Endpoint = laboratory-conrmed inuenza.
41 / 48
FLUVACS trial
42 / 48
42 / 48
FLUVACS trial
All 52 cases and 52 random controls were assayed for a
variety of markers (HAI, NAI, MN, AM titers,
proteins/virus/peptide magnitude/breadth).
Measured variables:
(EVERVAX)
43 / 48
All 52 cases and 52 random controls were assayed for a
variety of markers (HAI, NAI, MN, AM titers,
proteins/virus/peptide magnitude/breadth).
Measured variables:
(EVERVAX)
43 / 48
FLUVACS trial
All 52 cases and 52 random controls were assayed for a
variety of markers (HAI, NAI, MN, AM titers,
proteins/virus/peptide magnitude/breadth).
Measured variables:
(EVERVAX)
43 / 48
All 52 cases and 52 random controls were assayed for a
variety of markers (HAI, NAI, MN, AM titers,
proteins/virus/peptide magnitude/breadth).
Measured variables:
(EVERVAX)
43 / 48
FLUVACS trial
All 52 cases and 52 random controls were assayed for a
variety of markers (HAI, NAI, MN, AM titers,
proteins/virus/peptide magnitude/breadth).
Measured variables:
(EVERVAX)
43 / 48
All 52 cases and 52 random controls were assayed for a
variety of markers (HAI, NAI, MN, AM titers,
proteins/virus/peptide magnitude/breadth).
Measured variables:
(EVERVAX)
43 / 48
FLUVACS trial
All 52 cases and 52 random controls were assayed for a
variety of markers (HAI, NAI, MN, AM titers,
proteins/virus/peptide magnitude/breadth).
Measured variables:
(EVERVAX)
43 / 48
All 52 cases and 52 random controls were assayed for a
variety of markers (HAI, NAI, MN, AM titers,
proteins/virus/peptide magnitude/breadth).
Measured variables:
(EVERVAX)
43 / 48
FLUVACS trial
All 52 cases and 52 random controls were assayed for a
variety of markers (HAI, NAI, MN, AM titers,
proteins/virus/peptide magnitude/breadth).
Measured variables:
(EVERVAX)
43 / 48
All 52 cases and 52 random controls were assayed for a
variety of markers (HAI, NAI, MN, AM titers,
proteins/virus/peptide magnitude/breadth).
Measured variables:
(EVERVAX)
43 / 48
Variable sets
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
Variable sets
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
Variable sets
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
Variable sets
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
Variable sets
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
Variable sets
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
Variable sets
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
Variable sets
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
Variable sets
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
1.
2.3.4.5. Day 0 markers6. Day 30 markers7. Diff. markers8. (Day 0 + Day 30)9. (Day 0 + Diff.)
44 / 48
Analysis goals
We want to compare the quality of these nine sets of
variables for predicting u status in the placebo and
Fluzone arms separately.
We also want to compare the predictive quality of IgA, IgG,
and both IgA + IgG measurements.
We will use cross-validated Super Learning to do this.
45 / 48
We want to compare the quality of these nine sets of
variables for predicting u status in the placebo and
Fluzone arms separately.
We also want to compare the predictive quality of IgA, IgG,
and both IgA + IgG measurements.
We will use cross-validated Super Learning to do this.
45 / 48
Analysis goals
We want to compare the quality of these nine sets of
variables for predicting u status in the placebo and
Fluzone arms separately.
We also want to compare the predictive quality of IgA, IgG,
and both IgA + IgG measurements.
We will use cross-validated Super Learning to do this.
45 / 48
We want to compare the quality of these nine sets of
variables for predicting u status in the placebo and
Fluzone arms separately.
We also want to compare the predictive quality of IgA, IgG,
and both IgA + IgG measurements.
We will use cross-validated Super Learning to do this.
45 / 48
Analysis goals
We want to compare the quality of these nine sets of
variables for predicting u status in the placebo and
Fluzone arms separately.
We also want to compare the predictive quality of IgA, IgG,
and both IgA + IgG measurements.
We will use cross-validated Super Learning to do this.
45 / 48
We want to compare the quality of these nine sets of
variables for predicting u status in the placebo and
Fluzone arms separately.
We also want to compare the predictive quality of IgA, IgG,
and both IgA + IgG measurements.
We will use cross-validated Super Learning to do this.
45 / 48
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
EV x (Day 30 − Day 0) EV x (Day 0, Day 30) EV x (Day 0, Diff)
Day 30 − Day 0 EV x Day 0 EV x Day 30
Baseline Day 0 Day 30
0.4 0.6 0.8 1.00.4 0.6 0.8 1.00.4 0.6 0.8 1.0
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
AUC
Learner
Screen
lAll
screen.marginal.05
screen.marginal.10
l
l
l
l
Both
IgA
IgG
Neither 46 / 48
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
EV x (Day 30 − Day 0) EV x (Day 0, Day 30) EV x (Day 0, Diff)
Day 30 − Day 0 EV x Day 0 EV x Day 30
Baseline Day 0 Day 30
0.4 0.6 0.8 1.00.4 0.6 0.8 1.00.4 0.6 0.8 1.0
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
AUC
Learner
Screen
lAll
screen.marginal.05
screen.marginal.10
l
l
l
l
Both
IgA
IgG
Neither 46 / 48
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
EV x (Day 30 − Day 0) EV x (Day 0, Day 30) EV x (Day 0, Diff)
Day 30 − Day 0 EV x Day 0 EV x Day 30
Baseline Day 0 Day 30
0.20.40.60.81.0 0.20.40.60.81.0 0.20.40.60.81.0
SL
Discrete SL
SL.xgboost
SL
Discrete SL
SL.xgboost
SL
Discrete SL
SL.xgboost
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.xgboost
SL
Discrete SL
SL.xgboost
SL
Discrete SL
SL.glm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.gam
AUC
Learner
Screen
lAll
screen.marginal.05
screen.marginal.10
l
l
l
l
Both
IgA
IgG
Neither 47 / 48
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
EV x (Day 30 − Day 0) EV x (Day 0, Day 30) EV x (Day 0, Diff)
Day 30 − Day 0 EV x Day 0 EV x Day 30
Baseline Day 0 Day 30
0.20.40.60.81.0 0.20.40.60.81.0 0.20.40.60.81.0
SL
Discrete SL
SL.xgboost
SL
Discrete SL
SL.xgboost
SL
Discrete SL
SL.xgboost
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.xgboost
SL
Discrete SL
SL.xgboost
SL
Discrete SL
SL.glm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.gam
AUC
Learner
Screen
lAll
screen.marginal.05
screen.marginal.10
l
l
l
l
Both
IgA
IgG
Neither 47 / 48
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
EV x (Day 30 − Day 0) EV x (Day 0, Day 30) EV x (Day 0, Diff)
Day 30 − Day 0 EV x Day 0 EV x Day 30
Baseline Day 0 Day 30
0.40.60.81.0 0.40.60.81.0 0.40.60.81.0
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
AUC
Learner
Screen
lAll
screen.marginal.05
screen.marginal.10
l
l
l
l
Both
IgA
IgG
Neither 48 / 48
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
EV x (Day 30 − Day 0) EV x (Day 0, Day 30) EV x (Day 0, Diff)
Day 30 − Day 0 EV x Day 0 EV x Day 30
Baseline Day 0 Day 30
0.40.60.81.0 0.40.60.81.0 0.40.60.81.0
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
SL
Discrete SL
SL.glm
SL.bayesglm
SL.glmnet
SL.earth
SL.gam
SL.xgboost
SL.ranger
SL.mean
AUC
Learner
Screen
lAll
screen.marginal.05
screen.marginal.10
l
l
l
l
Both
IgA
IgG
Neither 48 / 48
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
EV x (Day 30 − Day 0) EV x (Day 0, Day 30) EV x (Day 0, Diff)
Day 30 − Day 0 EV x Day 0 EV x Day 30
Baseline Day 0 Day 30
0.20.40.60.8 0.20.40.60.8 0.20.40.60.8
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.ranger
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
AUC
Learner
l
l
l
l
Both
IgA
IgG
Neither 49 / 48
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
EV x (Day 30 − Day 0) EV x (Day 0, Day 30) EV x (Day 0, Diff)
Day 30 − Day 0 EV x Day 0 EV x Day 30
Baseline Day 0 Day 30
0.20.40.60.8 0.20.40.60.8 0.20.40.60.8
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.ranger
SL
Discrete SL
SL.bayesglm
SL
Discrete SL
SL.bayesglm
AUC
Learner
l
l
l
l
Both
IgA
IgG
Neither 49 / 48
Ferguson, T. S. (2014).Mathematical statistics: A decision theoretic
approach. Academic Press.
van der Vaart, A. W., Dudoit, S., and van der Laan, M. J. (2006). Oracle
inequalities for multi-fold cross validation.Statistics & Decisions,
24(3):351371.
Vapnik, V. (1992). Principles of risk minimization for learning theory. In
Advances in Neural Information Processing Systems, pages 831838.
Vapnik, V. (2013).The nature of statistical learning theory. Springer Science
& Business Media.
Vapnik, V. N. (1999). An overview of statistical learning theory.IEEE
Transactions on Neural Networks, 10(5):988999.
50 / 48
approach. Academic Press.
van der Vaart, A. W., Dudoit, S., and van der Laan, M. J. (2006). Oracle
inequalities for multi-fold cross validation.Statistics & Decisions,
24(3):351371.
Vapnik, V. (1992). Principles of risk minimization for learning theory. In
Advances in Neural Information Processing Systems, pages 831838.
Vapnik, V. (2013).The nature of statistical learning theory. Springer Science
& Business Media.
Vapnik, V. N. (1999). An overview of statistical learning theory.IEEE
Transactions on Neural Networks, 10(5):988999.
50 / 48
Tags
Categories
BusinessDownload
Download Slideshow Get the original presentation fileQuick Actions
Statistics
- Views 17
- Slides 181
- Age 82 days
Related Slideshows
1
DTI BPI Pivot Small Business - BUSINESS START UP PLAN
MeljunCortes
35 views
1
CATHOLIC EDUCATIONAL Corporate Responsibilities
MeljunCortes
36 views
11
Karin Schaupp – Evocation; lançamento: 2000
alfeuRIO
35 views
10
Pillars of Biblical Oneness in the Book of Acts
JanParon
30 views
31
7-10. STP + Branding and Product & Services Strategies.pptx
itsyash298
32 views
44
Business Legislation PPT - UNIT 1 jimllpkggg
slogeshk98
35 views