Deep Learning and Design Thinking

yenlung 6,349 views 117 slides May 09, 2018
Slide 1
Slide 1 of 194
Slide 1
1
Slide 2
2
Slide 3
3
Slide 4
4
Slide 5
5
Slide 6
6
Slide 7
7
Slide 8
8
Slide 9
9
Slide 10
10
Slide 11
11
Slide 12
12
Slide 13
13
Slide 14
14
Slide 15
15
Slide 16
16
Slide 17
17
Slide 18
18
Slide 19
19
Slide 20
20
Slide 21
21
Slide 22
22
Slide 23
23
Slide 24
24
Slide 25
25
Slide 26
26
Slide 27
27
Slide 28
28
Slide 29
29
Slide 30
30
Slide 31
31
Slide 32
32
Slide 33
33
Slide 34
34
Slide 35
35
Slide 36
36
Slide 37
37
Slide 38
38
Slide 39
39
Slide 40
40
Slide 41
41
Slide 42
42
Slide 43
43
Slide 44
44
Slide 45
45
Slide 46
46
Slide 47
47
Slide 48
48
Slide 49
49
Slide 50
50
Slide 51
51
Slide 52
52
Slide 53
53
Slide 54
54
Slide 55
55
Slide 56
56
Slide 57
57
Slide 58
58
Slide 59
59
Slide 60
60
Slide 61
61
Slide 62
62
Slide 63
63
Slide 64
64
Slide 65
65
Slide 66
66
Slide 67
67
Slide 68
68
Slide 69
69
Slide 70
70
Slide 71
71
Slide 72
72
Slide 73
73
Slide 74
74
Slide 75
75
Slide 76
76
Slide 77
77
Slide 78
78
Slide 79
79
Slide 80
80
Slide 81
81
Slide 82
82
Slide 83
83
Slide 84
84
Slide 85
85
Slide 86
86
Slide 87
87
Slide 88
88
Slide 89
89
Slide 90
90
Slide 91
91
Slide 92
92
Slide 93
93
Slide 94
94
Slide 95
95
Slide 96
96
Slide 97
97
Slide 98
98
Slide 99
99
Slide 100
100
Slide 101
101
Slide 102
102
Slide 103
103
Slide 104
104
Slide 105
105
Slide 106
106
Slide 107
107
Slide 108
108
Slide 109
109
Slide 110
110
Slide 111
111
Slide 112
112
Slide 113
113
Slide 114
114
Slide 115
115
Slide 116
116
Slide 117
117
Slide 118
118
Slide 119
119
Slide 120
120
Slide 121
121
Slide 122
122
Slide 123
123
Slide 124
124
Slide 125
125
Slide 126
126
Slide 127
127
Slide 128
128
Slide 129
129
Slide 130
130
Slide 131
131
Slide 132
132
Slide 133
133
Slide 134
134
Slide 135
135
Slide 136
136
Slide 137
137
Slide 138
138
Slide 139
139
Slide 140
140
Slide 141
141
Slide 142
142
Slide 143
143
Slide 144
144
Slide 145
145
Slide 146
146
Slide 147
147
Slide 148
148
Slide 149
149
Slide 150
150
Slide 151
151
Slide 152
152
Slide 153
153
Slide 154
154
Slide 155
155
Slide 156
156
Slide 157
157
Slide 158
158
Slide 159
159
Slide 160
160
Slide 161
161
Slide 162
162
Slide 163
163
Slide 164
164
Slide 165
165
Slide 166
166
Slide 167
167
Slide 168
168
Slide 169
169
Slide 170
170
Slide 171
171
Slide 172
172
Slide 173
173
Slide 174
174
Slide 175
175
Slide 176
176
Slide 177
177
Slide 178
178
Slide 179
179
Slide 180
180
Slide 181
181
Slide 182
182
Slide 183
183
Slide 184
184
Slide 185
185
Slide 186
186
Slide 187
187
Slide 188
188
Slide 189
189
Slide 190
190
Slide 191
191
Slide 192
192
Slide 193
193
Slide 194
194

About This Presentation

The presentation is an introduction to AI (deep learning). The key to success with AI is “asking good questions.” The talk was given in "Seminar in Information Systems and Applications" at National Tsing Hua University in Taiwan. During this talk, we discussed what a good question ...


Slide Content

Deep Learning
Yen-lung Tsai

Department of Mathematical Sciences
National Chengchi University
and
SeminarinInformationSystemsandApplications
Design Thinking
NTHU2018

!2
Askingagoodquestion
LearningNeuralNetworks
ApplyingDesignThinking
Outline

1
Askinga
GoodQuestion
!3

!4
f
Whatwewanttodoisjusttransforming thereal
worldproblemtoafunction.Thenweusedeep
learningtechniquestofindthefunction.

!5
Functionsaresolution
manuals

!6
Iwantoknowwhat
theanimalinthe
pictureis.

!7
Question Answer
Formosan
blackbear
Python
Wehavepartialanswers

!8
Quesiton Answer
Python
Theremightbeinfinitelymanypossible
casesthatwehaven’tseenbeore
?
Formosan
blackbear

!9
FindingourfunctionsbyNeuralNetworks!

!10
“Big3”ofDeepLearning
(NeuralWorks)
StandardNN CNN RNN

!11
AskingaQuestion
Seeingananimalinwild,wewantto
knowwhatisis?
1

!12
Transformingour
questionintoafunction
f
Formosan
blackbear
2

PreparingDataSetsfor
Training
( , )( ,“Python”),
,...xk+1,yk+1xk,ykx1,y1xn,yn
TrainingData TestingData
3
“Formosan 

blackbear”

!14
Reinforcement

Learning(RL)
GenerativeAdversarial

Network(GAN)
VAE
Capsule
StandardNN CNN RNN
ConstructingNN4

!15
Oncewedecideourstructureofneural
network,wehaveasetofparametersneedto
adjust.
θ={wi,bj}
Oncewedeterminetheparameters,wehavea
function:

!16
Learning
5
Thelearningissenttoourneuralnetworkusing
ourtrainingdata,adjustingourparameters,
andthenusingalossfunctiontoseehowmuch
differencewehaveandtheobservedvalue.

!17
Basically,weusethemethodcalled:
gradientdescent
ForNeuralNetworks,themethodisalsocalled:
backpropagation

!18
Iwanttoknowthe
closingpriceofa
stocktomorrow.

!19
f
Datex
theclosingpriceof
astockx

!20
f
Usethepricesofthe
previousweektopredict
thepriceforthenextday.
NN,CNN,RNN

!21
Iwanttoknowhow
mayhomersa
particular MLB
playercanhitin2018
season.

!22
Player
preformance
intheyearof
t-1
[Age,G,PA,AB,R,H,2B,3B,HR,
RBI,SB,BB,SO,OPS+,TB]
15features!
f
Numberof
homersinthe
yearoft
RNN

!23
Chatbot

!24
f
Currentword Nextword

!25
W1 W2
R1
EOS R1
R2
Rk
EOS
Notethatinthismode,eachinputandoutput
isnotoffixedlength!

!26
IwanttouseAIto
playgames(drivea
car,brewcoffee...)

!27
π
Thebest
action
CNN+NN

!28
Q
Score
ReinforcementLearning
+
Left

!29
somecharacters
aremissingina
fontIlike?>

!30
f
FontA FontB
CNN,VAE,GAN

2
AICoreTechnology-
Principlesof

NeuralNetworks
!31

Rememberwejusthaveto
learnafunction
!32
f
x1
x2
xn
y1
y2
ym

!33
ThreeStepsofLearning
Functions
Transferarealworldproblemintoafunction.
Collecttrainingmaterialsthatweknow"correct
answers."
Findthefunction!

BlackBoxLearning
Therearereallytechniquesforlearning
arbitraryfunctions
!34

neuralnetworks
!35

Inthe1980-1990orso,it
isaprettyfancystuff.
!36

!37
Hidden
Layer
Output
Layer
Input
Layer
BlackBox

Whatispowerfulisthatneural
networkswilllearneverything!
Andyoudon'thavetotellitwhatthe
functionshouldlooklike:linear,quadratic
polynomial,andsoon.
!38

Openthe
BlackBox!
!39

!40
FullyConnectedNeural
Networks
Verypopularsince1980s
StandardNN

!41

Theactionofeveryneuron
isbasicallythesame!
!42

!43
Eachneuronaccepts
severalinputsandthen
sendsoneoutput.

!44
Findtheweighted
sumofinputs.

!45
Plusbias.

!46
Applytheactivation
functiontotheweighted
sum.

!47
Popularactivationfunctions
ReLU Sigmoid Gaussian

!48
Parametersareweights,biases

!49
“Learned”neuralnetwork
1
2
1
21 1

!50
Supposewehavetheinput
21 1
(x1,x2)=(1,3)
1
3
1 3
8
1
2
1

!51
UsingReLUasactivationfunctions
1
3
1
2
1

!52
NNneedstobetrained
Using“previousexam
questions”totrainour
neuralnetwork
Learningmethodiscalled
backpropagation
TrainingMethods

When a neural network structure is determined
and activation functions are also determined,
it can be adjusted by weights, biases. We call
the set of these parameters θ, each of which
defines a function, and we treat it as a set.
TheFunctionSpaceofourNN
!53

Wearelookingfor
Makes theclosesttotheobjective
function
!54

Whatdoes“theclosest”mean
Thatmeansthevalueof “lossfunction”is
minimum
!55

Supposewehavethe
trainingdata
!56

!57
Wethevalueisassmallaspossible
Themostpopularloss
function
What’s
this!?

!58
x ???? e?1 ??vi
1 2 3 1
2 5 2 -3
3 6 8 2
4 7 2 -5
5 6 6 0
6 4 8 4
7 8 9 1
Howto
calculatetotal
error?

!59
x ???? e?1 ??vi
1 2 3 1
2 5 2 -3
3 6 8 2
4 7 2 -5
5 6 6 0
6 4 8 4
7 8 9 1
Total=0!!
Canwejust
addthemup?

!60
x ???? e?1 ??vi
1 2 3 1
2 5 2 -3
3 6 8 2
4 7 2 -5
5 6 6 0
6 4 8 4
7 8 9 1
Squared
Sum
1
9
4
25
0
16
1
56intotal

!61
Basically,dosuch
adjustment
learningrate

Whatdoesthis
terriblethingmean?
!62

RememberthatLisa
functionofw
1
,w
2
,b
1
,?>
!63
Thesmallerthebetter

Forsimplification,wefirst
thinkofLhasonlyone
variablew
!64

!65
Currentvalueofw
Howtogetto
theminimum?

!66
Currentvalueofw
Iknow,apointto
movetotheright!

!67
Howdoesthe
computer“see”it?

!68
Tangentisthekey!

!69
Tangentslope<0

!70
Tangentslope>0

Theslopeofthetangent
pointstothe(local)
maximumdirection!
!71

!72
Tangentslopeisthe
rateofchange
Tangentslope
Tangentslope

!73
Pointing(local)maximum
Tangentslope
Tangentslope

!74
Movingtoward(local)minimaKey
Inordertomakethevalueoftheloss
functionLsmaller,weshouldadjustthe
weightw:

!75
Supposecurrentw=a,weshouldupdateour
was:
Movingtoward(local)minimaKey

!76
AdjustingtheWeight
Tangentslope

!77
Sometimesit
willrunover!
Tangentslope

!78
Inordernottomaketoomuchadjustment
atonetime,wewillmultiplyasmall
numbercalledLeraningRate:
Movingtoward(local)minimaKey

!79
However,thereare
morethanone
parameter?>

!80
Example
Supposethat
Andwearemovingtowardtolocalminimum.

!81
Pretendtohaveonly
oneparameter!
say,itisw1

!82
Allparametersarefixed,exceptw1?>
One-variablefunction!
例⼦

!83
Similarly,wecanalsodefine
Example

!84
Sowecanadjustw
1
,w
2
,b
1
,andmakeL
smallerandsmaller?>
w
1
w
2
b
1
Example

!85
Writingtogetherseemsmorelearned!
Updated
Wecallthis the
gradientofL
Example

!86
PartialDerivativeDefinition
Whatwehavedoneisjustsocalledpartial
derivatives.

!87
Similarly,
PartialDerivativeDefinition

!88
gradientNotation
Recallthegradientof Lis:

!89
Weshouldhavecoolnotationforthe
gradient:
gradientNotation

!90
Sowecanupdateallparametersbythe
formula:
gradientNotation

This“learningmethod”hasa
fancyname
(SBEJFOU%FTDFOU
!91

!92
DeepLearning

Big3
StandardNN CNN RNN

!93
NoKidding!

!94
ConvolutionalNeuralNetwork
Superstarforimage
recognition
CNN

!95
Formosanblackbear
imagerecognition
f ( ) =

!96
PlayingVideoGames
Currentstates “thebest”actionπ ( ) =
reinforcementlearning

!97
filter 1
filter 2
input
?;? filter a?;?|?v?, wʇ??
?;???, Y5??bW??+?11?|?
v?E66@rP
ConvolutionalNeuralNetwork(CNN)

!98
ConvolutionalLayer
1

!99
Say, 3x3 filters
Weneedsome“filters”

!100
25525201
23404215
43135543
53450215
23111013
44115114
23220424
05453414
35
Thinkthisisanimage.
filter
Dot product
Thisislearned.
W=

!101
25525201
23404215
43135543
53450215
23111013
44115114
23220424
05453414
3527
filter
Move to the right
Samematrix
asbefore!
W=

!102
25525201
23404215
43135543
53450215
23111013
44115114
23220424
05453414
352744323638
363637363643
373723261735
292522181427
272524212432
313827342540
filter
All the way to the end
W=

!103
Neuronslooklikethis

!104
25525201
23404215
43135543
53450215
23111013
44115114
23220424
05453414
352744323638
363637363643
373723261735
292522181427
272524212432
313827342540
filter
Thepointsonthe
pictureareoneinput
layerneuron
W=

!105
25525201
23404215
43135543
53450215
23111013
44115114
23220424
05453414
352744323638
363637363643
373723261735
292522181427
272524212432
313827342540
filter
Convlayersalsoconsist
lotsofneural
W=

!106
25525201
23404215
43135543
53450215
23111013
44115114
23220424
05453414
352744323638
363637363643
373723261735
292522181427
272524212432
313827342540
filter
Thetwolayersarenotcompletelyconnected
W=

!107
25525201
23404215
43135543
53450215
23111013
44115114
23220424
05453414
352744323638
363637363643
373723261735
292522181427
272524212432
313827342540
filter
theweightsareshared

(sameaspreviousones)
W=

!108
352744323638
363637363643
373723261735
292522181427
272524212432
313827342540
Finally,wegeta6x6matrix,
andactuallyweusually
makeita8x8matrix.
Andwehavealotoffilters!

!109
Max-PoolingLayer
2

!110
We have to decide the size of area to have one
representative. Say, 2x2.
Basicallyitis“voting”

!111
352744323638
363637363643
373723261735
292522181427
272524212432
313827342540
36 44 43
37 26 35
38 34 40
“Vote”forthelargestnumber!!

!112
convolution, max-pooling, convolution, 

max-pooling…
Wecanrepeatandrepeat?>

!113
Afterwefinished,we sent
outputsto“normal”
neuralnetworks.

!114
RecurrentNeuralNetwork
NeuralNetoworkwith
memory
RNN

!115
Chatbot
Currentword Nextwordf ( ) =

!116
f
Currentword Nextword

!117
W1 W2
R1
EOS R1
R2
Rk
EOS
ChatbotApplication

!118
Infact,theinputdoesnot
havetobetext,butitisalso
possiblethatthevideo(one
byone)isoutput!The
outputcanstillbetext,and
themostcommonis
probablytoletthe
computersaywhat
happenedinthevideo.

!119
Translater
VideoCaptioningGenerator
ContextGenerator
Drawing
Applications

!120
AndrejKarpathyuseRNNtogenerateabookin
“Stacks”(adeeptopicinAlgebraicGeometry)
http://karpathy.github.io/2015/05/21/rnn-
effectiveness/

!121
PANDARUS:
Alas, I think he shall be come
approached and the day
When little srain would be attain'd into
being never fed,
And who is but a chain and subjects of
his death,
I should not sleep.
Second Senator:
They are away this miseries, produced
upon my soul,
Breaking and strongly should be
buried, when I perish
The earth and thoughts of many states.
“Shakespeare”

!122
Usually,outputsofaNeural
Networkarenotaffectedbythe
inputorders..

!123
However,RNNcellswilluseprevious
outputsaspartofinputs?>

!124
The“unfold”presentation.

!125
RecurrentNeuralNetwork(RNN)
ht1=σ(w1xt1+w2xt2+w3h3t−1+w4h4t−1+b1)
J?sH??z1
outputs T?az1
inputsrP

!126
Remark
Tomakeiteasierforeveryonetounderstand,we
willusesimplerrepresentations.Pleasenote
thattheinputsarevectorsandwillhave
respectedweight;whiletheoutputsarescalars.
xt=





xt,1
xt,2
.
.
.
xt,n





Lookslikethis

!127
Actuallylooklikethis
Remark

!128
h’sworksimilarly
Remark

!129
Eachcellofsome
layer,thereisone
output.
Inputscould
bevectors
Thek-thRNNCell

!130
TheconnectionsofaRNN
layer.NotethatRNNcells
willpasstheiroutputsto
otherRNNCellsofsame
layer.

!131
Formulaforthe
outputofa
standardRNNcell.

!132
LSTM
GRU
LongShortTermMemory
GatedRecurrentUnit
Whenwesay“RNN,”mostpeople
actuallythinkof?>

!133
LSTM
LongShortTermMemory
TheAceofRNNs

!134
k-th

LSTM
Cell
Cellstatus

!135
Gate
Important

!136
Theoutputisanumber
between0and1
sigmoid
Justdecidehowbigthe
“gate”willopen

!137
LSTMshasthree
typesofGates

!138
forgetgate inputgate outputgate

!139
Recall
tanh
sigmoid

!140
-1
1

!141
0
1
σ(x)=
1
1+e
−x

!142
k-th

LSTMcell
LSTMAgain
Thecellstateisfor
thecellonly
Theoutputofacell
will“share”with
othercells

!143
“New”cellstatus

!144

!145
Dowereallyneedtomake
thingssocomplicated?

!146
GRU
GatedRecurrentUnit
SimplifiedversionofLSTM

!147
Althoughithas“gated”inthename
Only2Gates

!148
resetgateupdategate

!149
We could ignore
previous inputs

!150

!151
Key NamesofRNNs
NowtalkingaboutRNN,infact,
includingtheoriginalRNN,LSTM,
GRUandothervariants.
Inparticular,theoriginalRNNis
calledVanillaRNN,andis
SimpleRNNinKeras.

3
Designthinking,
creativeproblem
solving
!152

!153
Empathize Define Ideate Prototype Test
DesignThinkingProcess
(Standfordd.school)
Veryclosetotheengineer’sapproach

!154
Ourmodelisrarelysuccessful
forthefirsttime.
—Yi-ShinChen,NationalTsingHuaUniversity

!155
MLBPlayerHomeRunsPrediction

!156
Yeart-1

[Age,G,PA,AB,R,H,2B,3B,HR,
RBI,SB,BB,SO,OPS+,TB]
15features
Numberof
homersinthe
yearoftf

!157
UseLSTM.Inputthedatainaperiodof10yearsand
predictthehomerunsinthenextyear.
OnlyoneLSTMlayer!

!158
Don'tguessthe
exactnumber,guess
whichinterval!
dividedinto5subintervals:

0-9,10-19,20-29,30-39,40+

!159
1
3
2
4
5
10-19
One-Hotencoding01000

⎣⎢⎢⎢⎢⎢⎢

⎦⎥⎥⎥⎥⎥⎥
0-9
10-19
20-29
30-39
40+

!160
MikeTrout(LAA)
Predicted30-39
Actual33
MookieBetts(BOS)
Predicted20-29
Actual24
JoseAltuve(HOU)
Predicted20-29
Actual24
KrisBryant(CHC)
Predicted30-39
Actual29
DanielMurphy(WSH)
Predicted20-29
Actual23
CoreySeager(LAD)
Predicted20-29
Actual22
2017forecastresult

!161
Fancywaystoask
questions!

!162
VAE
StandardNN CNN RNN
Reinforcement

Learning(RL)
GenerativeAdversarial

Network(GAN)
Capsule

!163
GenerativeAdversarial
Network
YannLeCunsaidGNNisthe
mostpromisingmodel
GAN

!164
There are many interesting recent
development in deep learning…
The most important one, in my opinion, is
adversarial training (also called GAN for
Generative Adversarial Networks). 


—YanLeCun(楊⽴昆),2016

!165
TheGANZoo
https://github.com/hindupuravinash/
the-gan-zoo

!166
generatorNoise
discriminator
RealorFake
AGANconsiststwoneuralnetworks,
aGeneratorandaDiscirminator.
z
x
G
D
G(z)

!167
Want
Closeto1
Want
Closeto0
Closeto1
D,GPK!
Discriminator
D
Generator
G

!168
Jun-YanZhuetal.(ECCV2016)
“Generative Visual Manipulation on the Natural Image Manifold”
iGAN
https://arxiv.org/abs/1609.03552

!169
Everyonecandraw!
https://youtu.be/9c4z6YsBGQ0

!170
Karrasetal.NVIDIAteam,(ICLR2018)
“Progressive Growing of GANs for Improved Quality, Stability, and Variation”
ProgressiveGAN
https://arxiv.org/abs/1710.10196

!171
Karras-Aila-Laine-Lehtinen
ByaNVIDIAteam
Theano,Python2,singleGPU(Hey,it’sNVIDIA)
ProgressiveGrowingofGANsforImproved
Quality,Stability,andVariation

!172
Thesearefake(1024x1024)

!173
Isola,Zhu,etal.,(CVPR2017)
“Image-to-Image Translation with Conditional Adversarial Networks”
Pix2Pix
https://arxiv.org/abs/1611.07004

!174
*FromtheoriginalpaperofIsola,Zhuetal. (2017)
Pix2pixtransferssatelliteimagestomaps

!175
Pix2pixgeneratesstreetviews
*FromtheoriginalpaperofIsola,Zhuetal. (2017)

!176
* byChristopherHesse
Pix2pixOn-lineVersion
https://affinelayer.com/pixsrv/

!177
Jun-YanZhuetal.(ICCV2017)
“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”
CycleGAN
https://arxiv.org/abs/1703.10593

!178
G

Generator
F

Generator
Domain
A
Domain
B
Discriminator
B
Discriminator
A
CycleGAN

!179
Don’tneedtopair
ourdatasets!!
Andopeninfinitelymanypossibilities...

!180
HorsestoZebras!!
https://youtu.be/9reHvktowLY

!181
OneexamplefromauthorsofCycleGAN

!182
Face-off(Tzer-JenWei)
https://youtu.be/Fea4kZq0oFQ

!183
ReinforcementLearning
AlphaGokeytechnique
ReinforcementLearning

!184
Agent

(computer)
Environmentstate
action
rewardrt

!185
Let’splay!

!186
π
Left
1PolicyBased
Right
or
State Action
policyfunction

!187
Usuallyitisnoteasyto
learndirectly...

!188
Q
Grade
+
Action
2ValueBased
(Usually
estimate
reward)
Valuefunction

!189
NetflixAlphaGoFilm(HighlyRecommended)

!190
Self-drivingcars

!191
Automatedtrading
system

!192
FixedoneETFeachtime
Startwith$20,000
Workforoneyear(SellallETFattheend)
Usingreinforcementlearning
*ETFdataprovidedbytheGlobal
Intelligence

!193
Datafrompastmonth
(20days,ina20x6
matrix) f
1
2
3
4
5
Buy20units
Buy10units
Notrading
Sell10units
Sell20units
5possibleactions

!194
CDQN Hold & buy CDQN Hold & buy
ETF1 17.71% 10.89% ETF11 10.76% 5.26%
ETF2 16.53% 12.6% ETF12 10.19% 13.17%
ETF3 16.3% 0.35% ETF13 7.8% 1.42%
ETF4 14.4% 13.25% ETF14 6.23% 3.56%
ETF5 14.3% 12.7% ETF15 5.73% 4.61%
ETF6 13.91% 13.37% ETF16 3.78% -12.76%
ETF7 13.17% 10.52% ETF17 2.85% 5.83%
ETF8 12.35% 17.07% ETF18 1.59% -4.45%
ETF9 11.68% 10.81% ETF19 1.07% -18.09%
ETF10 11.09% 8.14% ETF20 -0.59% -0.75%
TradingResults