AIS302-Artificial Neural Networks-Spr24-lec3.pdf

Nlp

1. AIS302
Artificial Neural
Networks
Spring 24
Ghada Khoriba
Associate Prof. of Artificial Intelligence
1

2. Agenda
2
References :
Princeton University COS 495 Instructor: Yingyu Liang
https://www.deeplearningbook.org/
Algorithmic Intelligence Laboratory, alinlab.kaist.ac.kr
•Recap
•RNN, Recurrent neural networks

3. Recurrentneuralnetworks
•Datesbackto(Rumelhartetal.,1986)
•Afamilyofneuralnetworksforhandlingsequentialdata,which
involvesvariable lengthinputsor outputs
•Especially,fornaturallanguageprocessing(NLP)

4. Sequentialdata
•Eachdatapoint:Asequenceof vectors!("),for1≤$≤%
•Batchdata:manysequenceswithdifferentlengths%
•Label:canbea scalar,a vector,orevena sequence
•Example
•Sentimentanalysis
•Machinetranslation

5. Example:machinetranslation

6. Morecomplicatedsequential data
•Datapoint:twodimensionalsequenceslikeimages
•Label:differenttypeofsequencesliketextsentences
•Example:image captioning

7. Imagecaptioning
Figurefrom thepaper“DenseCap:FullyConvolutionalLocalizationNetworksforDenseCaptioning”,
by JustinJohnson,AndrejKarpathy,LiFei-Fei

8. 4
Sequential Data Problems
Fixed-sized
input
to fixed-sized
output
(e.g. image
classification)
Sequence output
(e.g. image captioning
takes an image and
outputs a sentence of
words).
Sequence input(e.g.
sentiment analysis
where a given sentence
is classified as
expressing positive or
negative sentiment).
Sequence input and
sequence output(e.g.
Machine Translation: an
RNN reads a sentence in
English and then outputs
a sentence in French)
Synced sequence input
and output (e.g. video
classification where we
wish to label each frame
of the video)
Credits: Andrej Karpathy

9. Atypicaldynamicsystem
!("+1)=#(!";&)
FigurefromDeepLearning,
Goodfellow,BengioandCourville

10. Asystem driven byexternaldata
!("+1)=#(!",)("+1);&)

11. Compactview
!("+1)=#(!",)("+1);&)

12. Compactview
!("+1)=#(!",)("+1);&)
Key:thesame&and '
for alltimesteps
square:onesteptimedelay

13. Recurrentneuralnetworks
•Usethesamecomputationalfunctionandparametersacrossdifferent
timestepsofthesequence.
•Eachtimestep:takestheinputentryandtheprevioushiddenstateto
computethe outputentry
•Loss:typicallycomputedeverytimestep

14. Recurrentneuralnetworks
Label
Loss
Output
State
Input

15. Recurrentneuralnetworks
Mathformula:

16. Advantage
•Hiddenstate:alossysummaryof thepast
•Sharedfunctionsandparameters:greatlyreducethecapacity and
goodforgeneralizationinlearning
•Explicitly usethepriorknowledgethatthesequentialdatacanbe
processedbyinthesamewayatdifferenttimestep(e.g.,NLP)

17. Advantage
•Hiddenstate:alossysummaryof thepast
•Sharedfunctionsandparameters:greatlyreducethecapacityand
goodforgeneralizationinlearning
•Explicitly usethepriorknowledgethatthesequentialdatacanbe
processedbyinthesamewayatdifferenttimestep(e.g.,NLP)

18. TrainingRNN
•Principle:unfoldthecomputationalgraphandusebackpropagation
•Calledback-propagationthroughtime(BPTT)algorithm
•Canthenapplyanygeneral-purposegradient-basedtechniques

19. TrainingRNN
•Principle:unfoldthecomputationalgraph,andusebackpropagation
•Calledback-propagationthroughtime(BPTT)algorithm
•Canthenapplyanygeneral-purposegradient-basedtechniques
•Conceptually:firstcomputethegradientsoftheinternalnodes,then
computethe gradientsofthe parameters

20. Recurrentneuralnetworks
Mathformula:
FigurefromDeepLearning,
Goodfellow,BengioandCourville

21. Recurrentneuralnetworks
Gradientat!("):(total
lossissumofthoseat
differenttimesteps)
FigurefromDeepLearning,
Goodfellow,BengioandCourville

22. Recurrentneuralnetworks
Gradient at"("):

23. Recurrentneuralnetworks
Gradientat#($):
is the gradient of the loss L with
respect to the output O at time
step τ
gradients are propagated backwards through time to update the parameters
of the RNN. This allows the RNN to learn temporal dependencies from
sequences of data.

24. Recurrentneuralnetworks
Gradientat #("):

25. Recurrentneuralnetworks
Gradientatparameter$:

27. RNN
•Usethesamecomputationalfunctionandparametersacrossdifferent
timestepsofthesequence
•Eachtimestep:takestheinputentryandtheprevioushiddenstateto
computethe outputentry
•Loss:typicallycomputedeverytimestep
•Manyvariants
•Informationaboutthepastcan beinmany otherforms
•Onlyoutputattheendof thesequence

29. Example:onlyoutputat the
end

30. BidirectionalRNNs
•Manyapplications:outputattime!maydependonthewholeinput
sequence
•Exampleinspeechrecognition:correctinterpretationofthecurrent
soundmaydependonthenextfewphonemes,potentiallyeventhe
nextfewwords
•BidirectionalRNNsareintroducedtoaddressthis

31. BiRNNs