Machine Learning- Perceptron_Backpropogation_Module 3.pdf
DrShivashankar1
1,090 views
37 slides
Jul 19, 2024
Slide 1 of 37
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
About This Presentation
Machine Learning- Perceptron_Backpropogation
Slide Content
MACHINE LEARNING (INTEGRATED)
(21ISE62)
Dr. Shivashankar
Professor
Department of Information Science & Engineering
GLOBAL ACADEMY OF TECHNOLOGY-Bengaluru
7/19/2024 1Dr. Shivashankar, ISE, GAT
GLOBAL ACADEMY OF TECHNOLOGY
Ideal Homes Township, RajarajeshwariNagar, Bengaluru –560 098
Department of Information Science & Engineering
(21ISE62)
Dr. Shivashankar
Professor
Department of Information Science & Engineering
GLOBAL ACADEMY OF TECHNOLOGY-Bengaluru
7/19/2024 1Dr. Shivashankar, ISE, GAT
GLOBAL ACADEMY OF TECHNOLOGY
Ideal Homes Township, RajarajeshwariNagar, Bengaluru –560 098
Department of Information Science & Engineering
Course Outcomes
AfterCompletionofthecourse,studentwillbeableto:
IllustrateRegressionTechniquesandDecisionTreeLearning
Algorithm.
ApplySVM,ANNandKNNalgorithmtosolveappropriateproblems.
ApplyBayesianTechniquesandderiveeffectivelearningrules.
IllustrateperformanceofAIandMLalgorithmsusingevaluation
techniques.
Understandreinforcementlearninganditsapplicationinrealworld
problems.
TextBook:
1.TomM.Mitchell,MachineLearning,McGrawHillEducation,IndiaEdition2013.
2.EthemAlpaydın,Introductiontomachinelearning,MITpress,Secondedition.
3.Pang-NingTan,MichaelSteinbach,VipinKumar,IntroductiontoDataMining,
Pearson,FirstImpression,2014.
7/19/2024 2Dr. Shivashankar, ISE, GAT
AfterCompletionofthecourse,studentwillbeableto:
IllustrateRegressionTechniquesandDecisionTreeLearning
Algorithm.
ApplySVM,ANNandKNNalgorithmtosolveappropriateproblems.
ApplyBayesianTechniquesandderiveeffectivelearningrules.
IllustrateperformanceofAIandMLalgorithmsusingevaluation
techniques.
Understandreinforcementlearninganditsapplicationinrealworld
problems.
TextBook:
1.TomM.Mitchell,MachineLearning,McGrawHillEducation,IndiaEdition2013.
2.EthemAlpaydın,Introductiontomachinelearning,MITpress,Secondedition.
3.Pang-NingTan,MichaelSteinbach,VipinKumar,IntroductiontoDataMining,
Pearson,FirstImpression,2014.
7/19/2024 2Dr. Shivashankar, ISE, GAT
Module 3: Artificial Neural Networks
•Motivationbehindneuralnetworkishumanbrain.Humanbrainiscalledasthebestprocessor
eventhoughitworksslowerthanothercomputers.
•Humanbraincells,calledneurons,formacomplex,highlyinterconnectednetworkandsend
electricalsignalstoeachothertohelphumansprocessinformation.
•Similarly,anartificialneuralnetworkismadeofartificialneuronsthatworktogethertosolvea
realworldproblems.
•Artificialneuronsaresoftwaremodules,callednodes,andartificialneuralnetworksaresoftware
programsoralgorithmsthat,attheircore,usecomputingsystemstosolvemathematical
calculations.
7/19/2024 3Dr. Shivashankar, ISE, GAT
Fig 3.3: Artificial Neural Networks
•Motivationbehindneuralnetworkishumanbrain.Humanbrainiscalledasthebestprocessor
eventhoughitworksslowerthanothercomputers.
•Humanbraincells,calledneurons,formacomplex,highlyinterconnectednetworkandsend
electricalsignalstoeachothertohelphumansprocessinformation.
•Similarly,anartificialneuralnetworkismadeofartificialneuronsthatworktogethertosolvea
realworldproblems.
•Artificialneuronsaresoftwaremodules,callednodes,andartificialneuralnetworksaresoftware
programsoralgorithmsthat,attheircore,usecomputingsystemstosolvemathematical
calculations.
7/19/2024 3Dr. Shivashankar, ISE, GAT
Fig 3.3: Artificial Neural Networks
Conti..
InputLayer
•Thisisthefirstlayerinatypicalneuralnetwork.
•Inputlayerneuronsreceivetheinputinformationfromtheoutsideworldenterstheartificialneural
network,processitthroughamathematicalfunction(activationfunction),andtransmitoutputtothe
nextlayer’sneuronsbasedoncomparisonwithapresetthresholdvalue.
•Wepre-processtext,image,audio,video,andothertypesofdatatoderivetheirnumericrepresentation.
HiddenLayer
•Hiddenlayerstaketheirinputfromtheinputlayerorotherhiddenlayersandhavealargenumberof
hiddenlayers.Itcontainsthesummationandactivationfunction.
•Eachhiddenlayeranalyzestheoutputfromthepreviouslayer,processesitfurther,andpassesitonto
thenextlayer.Herealso,wemultiplythedatabyedgeweightsasitistransmittedtothenextlayer.
OutputLayer
•Theoutputlayergivesthefinalresultofallthedataprocessingbytheartificialneuralnetwork.Itcan
havesingleormultiplenodes.
•Forinstance,ifwehaveabinary(yes/no)classificationproblem,theoutputlayerwillhaveoneoutput
node,whichwillgivetheresultas1or0.
•However,ifwehaveamulti-classclassificationproblem,theoutputlayermightconsistofmorethanone
outputnode.
7/19/2024 4Dr. Shivashankar, ISE, GAT
InputLayer
•Thisisthefirstlayerinatypicalneuralnetwork.
•Inputlayerneuronsreceivetheinputinformationfromtheoutsideworldenterstheartificialneural
network,processitthroughamathematicalfunction(activationfunction),andtransmitoutputtothe
nextlayer’sneuronsbasedoncomparisonwithapresetthresholdvalue.
•Wepre-processtext,image,audio,video,andothertypesofdatatoderivetheirnumericrepresentation.
HiddenLayer
•Hiddenlayerstaketheirinputfromtheinputlayerorotherhiddenlayersandhavealargenumberof
hiddenlayers.Itcontainsthesummationandactivationfunction.
•Eachhiddenlayeranalyzestheoutputfromthepreviouslayer,processesitfurther,andpassesitonto
thenextlayer.Herealso,wemultiplythedatabyedgeweightsasitistransmittedtothenextlayer.
OutputLayer
•Theoutputlayergivesthefinalresultofallthedataprocessingbytheartificialneuralnetwork.Itcan
havesingleormultiplenodes.
•Forinstance,ifwehaveabinary(yes/no)classificationproblem,theoutputlayerwillhaveoneoutput
node,whichwillgivetheresultas1or0.
•However,ifwehaveamulti-classclassificationproblem,theoutputlayermightconsistofmorethanone
outputnode.
7/19/2024 4Dr. Shivashankar, ISE, GAT
Conti..
•Itisusuallyacomputationalnetworkbasedonbiologicalneuralnetworksthat
constructthestructureofthehumanbrain.
•Similartoahumanbrainhasneuronsinterconnectedtoeachother,artificial
neuralnetworksalsohaveneuronsthatarelinkedtoeachotherinvarious
layersofthenetworks.
•Theseneuronsareknownasnodes.
•Artificialneuralnetworks(ANNs)provideageneral,practicalmethodfor
learningreal-valued,discrete-valued,andvector-valuedfunctionsfrom
examples.
•ANNlearningisrobusttoerrorsinthetrainingdataandhasbeensuccessfully
appliedtoproblemssuchasinterpretingvisualscenes,speechrecognition,
andlearningrobotcontrolstrategies.
•Thefastestneuronswitchingtimesareknowntobeontheorderof10
−3
seconds--quiteslowcomparedtocomputerswitchingspeedsof10
−10
seconds.
7/19/2024 5Dr. Shivashankar, ISE, GAT
•Itisusuallyacomputationalnetworkbasedonbiologicalneuralnetworksthat
constructthestructureofthehumanbrain.
•Similartoahumanbrainhasneuronsinterconnectedtoeachother,artificial
neuralnetworksalsohaveneuronsthatarelinkedtoeachotherinvarious
layersofthenetworks.
•Theseneuronsareknownasnodes.
•Artificialneuralnetworks(ANNs)provideageneral,practicalmethodfor
learningreal-valued,discrete-valued,andvector-valuedfunctionsfrom
examples.
•ANNlearningisrobusttoerrorsinthetrainingdataandhasbeensuccessfully
appliedtoproblemssuchasinterpretingvisualscenes,speechrecognition,
andlearningrobotcontrolstrategies.
•Thefastestneuronswitchingtimesareknowntobeontheorderof10
−3
seconds--quiteslowcomparedtocomputerswitchingspeedsof10
−10
seconds.
7/19/2024 5Dr. Shivashankar, ISE, GAT
Biological Motivation
•Theterm"ArtificialNeuralNetwork(ANN)"referstoabiologicallyinspiredsub-fieldof
artificialintelligencemodeledafterthebrain.
•ANNhasbeeninspiredbybiologicallearningsystembiologicallearningsystemismade
upofcomplexwebofinterconnectedneurons.
•ArtificialinterconnectedneuronslikebiologicalneuronsmakingupanANN.
•Eachbiologicalneuroniscapableoftakinganumberofinputsandproduceoutput.
•OnemotivationforANNisthattoworkforaparticulartaskidentificationthroughmany
parallelprocesses.
Considerhumanbrain:
•Numberofneurons~10
11
neurons
•Connectionsperneurons~10
4−5
•Neuronsswitchingtime~10
−3
seconds(0.001)
•Computerswitchingtime~10
−10
seconds
•Scenerecognitiontime~10
−1
seconds(0.1)
7/19/2024 6Dr. Shivashankar, ISE, GAT
•Theterm"ArtificialNeuralNetwork(ANN)"referstoabiologicallyinspiredsub-fieldof
artificialintelligencemodeledafterthebrain.
•ANNhasbeeninspiredbybiologicallearningsystembiologicallearningsystemismade
upofcomplexwebofinterconnectedneurons.
•ArtificialinterconnectedneuronslikebiologicalneuronsmakingupanANN.
•Eachbiologicalneuroniscapableoftakinganumberofinputsandproduceoutput.
•OnemotivationforANNisthattoworkforaparticulartaskidentificationthroughmany
parallelprocesses.
Considerhumanbrain:
•Numberofneurons~10
11
neurons
•Connectionsperneurons~10
4−5
•Neuronsswitchingtime~10
−3
seconds(0.001)
•Computerswitchingtime~10
−10
seconds
•Scenerecognitiontime~10
−1
seconds(0.1)
7/19/2024 6Dr. Shivashankar, ISE, GAT
NEURAL NETWORK REPRESENTATIONS
•Inanartificialneuralnetwork,aneuroneisalogisticunit.
Feedinputviainputwires.
Logisticunitdoescomputation.
Sendsoutputdownoutputwires
•Thatlogisticcomputationisjustlikeourpreviouslogisticregressionhypothesis
calculation.
•Input–30*32grid–camera.
•Output–Vehicleissteered.
•Training–Observingsteeringcommandsofhumandrivingthevehicle.
•960inputs–30outputunits–Steeringcommandrecommendedmost.
•ALVINN–acyclicgraph.
7/19/2024 7Dr. Shivashankar, ISE, GAT
•Inanartificialneuralnetwork,aneuroneisalogisticunit.
Feedinputviainputwires.
Logisticunitdoescomputation.
Sendsoutputdownoutputwires
•Thatlogisticcomputationisjustlikeourpreviouslogisticregressionhypothesis
calculation.
•Input–30*32grid–camera.
•Output–Vehicleissteered.
•Training–Observingsteeringcommandsofhumandrivingthevehicle.
•960inputs–30outputunits–Steeringcommandrecommendedmost.
•ALVINN–acyclicgraph.
7/19/2024 7Dr. Shivashankar, ISE, GAT
PERCEPTRONS
•OnetypeofANNsystemisbasedonaunitcalledaperceptron.
•Aperceptrontakesavectorofreal-valuedinputs,calculatesalinearcombinationoftheseinputs,
thenoutputsa1iftheresultisgreaterthansomethresholdand-1otherwise.
•Moreprecisely,giveninputs�
1through�
2,theoutputo(�
1,……�
�)computedbytheperceptron
is o(�
1,……�
�)=ቊ
1���
0+�
1�
1+�
2�
2+⋯+�
��
�>0
−1��ℎ������
•whereeach�
�isareal-valuedconstant,orweight,thatdeterminesthecontributionofinput�
�to
theperceptronoutput.
•Wewillsometimeswritetheperceptronfunctionas
�(Ԧ�)=sgn�.Ԧ�
•Where,sgn(y)=ቊ
1���>0
−1��ℎ������
•Learningaperceptroninvolveschoosingvaluesfortheweights�
0,…..�
�.Therefore,thespaceH
ofcandidatehypothesesconsideredinperceptronlearningisthesetofallpossiblereal-valued
weightvectors.
•H=�|�????????????
�+1
7/19/2024 8Dr. Shivashankar, ISE, GAT
•OnetypeofANNsystemisbasedonaunitcalledaperceptron.
•Aperceptrontakesavectorofreal-valuedinputs,calculatesalinearcombinationoftheseinputs,
thenoutputsa1iftheresultisgreaterthansomethresholdand-1otherwise.
•Moreprecisely,giveninputs�
1through�
2,theoutputo(�
1,……�
�)computedbytheperceptron
is o(�
1,……�
�)=ቊ
1���
0+�
1�
1+�
2�
2+⋯+�
��
�>0
−1��ℎ������
•whereeach�
�isareal-valuedconstant,orweight,thatdeterminesthecontributionofinput�
�to
theperceptronoutput.
•Wewillsometimeswritetheperceptronfunctionas
�(Ԧ�)=sgn�.Ԧ�
•Where,sgn(y)=ቊ
1���>0
−1��ℎ������
•Learningaperceptroninvolveschoosingvaluesfortheweights�
0,…..�
�.Therefore,thespaceH
ofcandidatehypothesesconsideredinperceptronlearningisthesetofallpossiblereal-valued
weightvectors.
•H=�|�????????????
�+1
7/19/2024 8Dr. Shivashankar, ISE, GAT
Representational Power of Perceptron
•Wecanviewtheperceptronasrepresentingahyperplanedecisionsurfaceinthen-
dimensionalspaceofinstances(i.e.,points).
•Theperceptronoutputsa1forinstanceslyingononesideofthehyperplaneand
outputsanda-1forinstanceslyingontheotherside.
•Theequationforthisdecisionhyperplaneis�.Ԧ�=0.
•Ofcourse,somesetsofpositiveandnegativeexamplescannotbeseparatedbyany
hyperplane.
•Thosethatcanbeseparatedarecalledlinearlyseparablesetsofexamples.
•Asingleperceptroncanbeusedtorepresentmanybooleanfunctions.
7/19/2024 9Dr. Shivashankar, ISE, GAT
•Wecanviewtheperceptronasrepresentingahyperplanedecisionsurfaceinthen-
dimensionalspaceofinstances(i.e.,points).
•Theperceptronoutputsa1forinstanceslyingononesideofthehyperplaneand
outputsanda-1forinstanceslyingontheotherside.
•Theequationforthisdecisionhyperplaneis�.Ԧ�=0.
•Ofcourse,somesetsofpositiveandnegativeexamplescannotbeseparatedbyany
hyperplane.
•Thosethatcanbeseparatedarecalledlinearlyseparablesetsofexamples.
•Asingleperceptroncanbeusedtorepresentmanybooleanfunctions.
7/19/2024 9Dr. Shivashankar, ISE, GAT
Cont…
•ANDandORcanbeviewedasspecialcasesofm-of-nfunctions:thatis,functions
whereatleastmoftheninputstotheperceptronmustbetrue.
•TheORfunctioncorrespondstom=1andtheANDfunctiontom=n.
•Anym-of-nfunctioniseasilyrepresentedusingaperceptronbysettingallinput
weightstothesamevalue(e.g.,0.5)andthensettingthethresholdtaccordingly.
•PerceptroncanrepresentalloftheprimitivebooleanfunctionsAND,OR,NAND(¬
AND),andNOR(¬OR).
•TheabilityofperceptrontorepresentAND,OR,NAND,andNORisimportantbecause
everybooleanfunctioncanberepresentedbysomenetworkofinterconnectedunits
basedontheseprimitives.
7/19/2024 10Dr. Shivashankar, ISE, GAT
•ANDandORcanbeviewedasspecialcasesofm-of-nfunctions:thatis,functions
whereatleastmoftheninputstotheperceptronmustbetrue.
•TheORfunctioncorrespondstom=1andtheANDfunctiontom=n.
•Anym-of-nfunctioniseasilyrepresentedusingaperceptronbysettingallinput
weightstothesamevalue(e.g.,0.5)andthensettingthethresholdtaccordingly.
•PerceptroncanrepresentalloftheprimitivebooleanfunctionsAND,OR,NAND(¬
AND),andNOR(¬OR).
•TheabilityofperceptrontorepresentAND,OR,NAND,andNORisimportantbecause
everybooleanfunctioncanberepresentedbysomenetworkofinterconnectedunits
basedontheseprimitives.
7/19/2024 10Dr. Shivashankar, ISE, GAT
The Perceptron Training Rule
•learningproblemistodetermineaweightvectorthatcausestheperceptronto
producethecorrectoutputforeachofthegiventrainingexamples.
•Onewaytolearnanacceptableweightvectoristobeginwithrandomweights,then
iterativelyapplytheperceptrontoeachtrainingexample,modifyingtheperceptron
weightswheneveritmisclassifiesanexample.
•Thisprocessisrepeated,iteratingthroughthetrainingexamplesasmanytimesas
neededuntiltheperceptronclassifiesalltrainingexamplescorrectly.
•Ateverystepoffeedingatrainingexample,whentheperceptronfailstoproducethe
correct+1/-1,wereviseeveryweight�
�associatedwitheveryinput�
�,accordingto
thefollowingrule:
�
�←�
�+∆�
�
Where ∆�
�=ƞ�−��
�
tisthetargetoutputforthecurrenttrainingexample,
oistheoutputgeneratedbytheperceptron,and
ƞisapositiveconstantcalledthelearningrate.Theroleofthelearningrateistomoderate
thedegreetowhichweightsarechangedateachstep.
∆:Thisisthelearningrate,orthestepsize.
7/19/2024 11Dr. Shivashankar, ISE, GAT
•learningproblemistodetermineaweightvectorthatcausestheperceptronto
producethecorrectoutputforeachofthegiventrainingexamples.
•Onewaytolearnanacceptableweightvectoristobeginwithrandomweights,then
iterativelyapplytheperceptrontoeachtrainingexample,modifyingtheperceptron
weightswheneveritmisclassifiesanexample.
•Thisprocessisrepeated,iteratingthroughthetrainingexamplesasmanytimesas
neededuntiltheperceptronclassifiesalltrainingexamplescorrectly.
•Ateverystepoffeedingatrainingexample,whentheperceptronfailstoproducethe
correct+1/-1,wereviseeveryweight�
�associatedwitheveryinput�
�,accordingto
thefollowingrule:
�
�←�
�+∆�
�
Where ∆�
�=ƞ�−��
�
tisthetargetoutputforthecurrenttrainingexample,
oistheoutputgeneratedbytheperceptron,and
ƞisapositiveconstantcalledthelearningrate.Theroleofthelearningrateistomoderate
thedegreetowhichweightsarechangedateachstep.
∆:Thisisthelearningrate,orthestepsize.
7/19/2024 11Dr. Shivashankar, ISE, GAT
The Perceptron Training Rule
•InordertotrainthePerceptronf(X<W):
�
�←�
�+∆�
�
Where ∆�
�=ƞ�−��
�
7/19/2024 12Dr. Shivashankar, ISE, GAT
Initialize the weight, W, randomly.
For as many times as necessary:
For each training examples x????????????
Compute f(x,W)
If x is misclassified:
Modify the weight, �
�associated with every �
�in x.
•InordertotrainthePerceptronf(X<W):
�
�←�
�+∆�
�
Where ∆�
�=ƞ�−��
�
7/19/2024 12Dr. Shivashankar, ISE, GAT
Initialize the weight, W, randomly.
For as many times as necessary:
For each training examples x????????????
Compute f(x,W)
If x is misclassified:
Modify the weight, �
�associated with every �
�in x.
Problem
Problem1:Assume�
1=0.6����
2=0.6,�ℎ���ℎ�??????�=1���??????�����������ƞ=0.5.
ComputeORgateusingperceptrontrainingrule.
Solution:1.A=0,B=0andtarget=0
�
��
�=�
1�
1+�
2�
2
=0.6*0+0.6*0=0
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0
2.A=0,B=1andtarget=1
�
��
�=0.6∗0+0.6*1=0.6
Thisisnotgreaterthanthethresholdvalueof1.Sotheoutput=0.
�
�=�
�+ƞ(t-o)�
�
�
1=0.6+0.5(1-0)0=0.6
�
2=0.6+0.5(1-0)1=1.1
Now�
�=0.6,�
�=1.1,threshold=1andlearningrateƞ=0.5
7/19/2024 13Dr. Shivashankar, ISE, GAT
A B Y=A+B
(Target)
0 0 0
0 1 1
1 0 1
1 1 1
Problem1:Assume�
1=0.6����
2=0.6,�ℎ���ℎ�??????�=1���??????�����������ƞ=0.5.
ComputeORgateusingperceptrontrainingrule.
Solution:1.A=0,B=0andtarget=0
�
��
�=�
1�
1+�
2�
2
=0.6*0+0.6*0=0
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0
2.A=0,B=1andtarget=1
�
��
�=0.6∗0+0.6*1=0.6
Thisisnotgreaterthanthethresholdvalueof1.Sotheoutput=0.
�
�=�
�+ƞ(t-o)�
�
�
1=0.6+0.5(1-0)0=0.6
�
2=0.6+0.5(1-0)1=1.1
Now�
�=0.6,�
�=1.1,threshold=1andlearningrateƞ=0.5
7/19/2024 13Dr. Shivashankar, ISE, GAT
A B Y=A+B
(Target)
0 0 0
0 1 1
1 0 1
1 1 1
Problem
•Now�
�=0.6,�
�=1.1,threshold=1andlearningrateƞ=0.5
1.A=0,B=0andtarget=0
�
��
�=�
1�
1+�
2�
2
=0.6*0+1.1*0=0
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0
2.A=0,B=1andtarget=1
�
��
�=0.6∗0+1.1*1=1.1
Thisisgreaterthanthethresholdvalueof1.
Sotheoutput=1.
3.A=1,B=0andtarget=1
�
��
�=0.6∗1+1.1*0=0.6
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0.
�
�=�
�+ƞ(t-0)�
�
�
1=0.6+0.5(1-0)1=1.1
�
2=1.1+0.5(1-0)0=1.1
7/19/2024 14Dr. Shivashankar, ISE, GAT
•Now�
�=0.6,�
�=1.1,threshold=1andlearningrateƞ=0.5
1.A=0,B=0andtarget=0
�
��
�=�
1�
1+�
2�
2
=0.6*0+1.1*0=0
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0
2.A=0,B=1andtarget=1
�
��
�=0.6∗0+1.1*1=1.1
Thisisgreaterthanthethresholdvalueof1.
Sotheoutput=1.
3.A=1,B=0andtarget=1
�
��
�=0.6∗1+1.1*0=0.6
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0.
�
�=�
�+ƞ(t-0)�
�
�
1=0.6+0.5(1-0)1=1.1
�
2=1.1+0.5(1-0)0=1.1
7/19/2024 14Dr. Shivashankar, ISE, GAT
Problem
•Now�
�=1.1,�
�=1.1,threshold=1andlearningrateƞ=0.5
1.A=0,B=0andtarget=0
�
��
�=�
1�
1+�
2�
2
=1.1*0+1.1*0=0
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0
2.A=0,B=1andtarget=1
�
��
�=1.1∗0+1.1*1=1.1
Thisisgreaterthanthethresholdvalueof1.
Sotheoutput=1.
3.A=1,B=0andtarget=1
�
��
�=1.1∗1+1.1*0=1.1
Thisisgreaterthanthethresholdvalueof1.
Sotheoutput=1.
4.A=1.B=1andtarget=1
�
�=1.1*1+1.1*1=2.2
Thisisgreaterthanthethresholdvalueof1.
Sotheoutput=1.
7/19/2024 15Dr. Shivashankar, ISE, GAT
B
A
1.1
1.1
∈ ??????=1
??????�����
•Now�
�=1.1,�
�=1.1,threshold=1andlearningrateƞ=0.5
1.A=0,B=0andtarget=0
�
��
�=�
1�
1+�
2�
2
=1.1*0+1.1*0=0
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0
2.A=0,B=1andtarget=1
�
��
�=1.1∗0+1.1*1=1.1
Thisisgreaterthanthethresholdvalueof1.
Sotheoutput=1.
3.A=1,B=0andtarget=1
�
��
�=1.1∗1+1.1*0=1.1
Thisisgreaterthanthethresholdvalueof1.
Sotheoutput=1.
4.A=1.B=1andtarget=1
�
�=1.1*1+1.1*1=2.2
Thisisgreaterthanthethresholdvalueof1.
Sotheoutput=1.
7/19/2024 15Dr. Shivashankar, ISE, GAT
B
A
1.1
1.1
∈ ??????=1
??????�����
Problem
Problem2:Assume�
1=1.2����
2=0.6,�ℎ���ℎ�??????�=1���??????�����������ƞ=0.5.
ComputeANDgateusingperceptrontrainingrule.
Solution:1.A=0,B=0andtarget=0
�
��
�=�
1�
1+�
2�
2
=1.2*0+0.6*0=0
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0
2.A=0,B=1andtarget=0
�
��
�=1.2∗0+0.6*1=0.6
Thisisnotgreaterthanthethresholdvalueof1,Sotheoutput=0.
3.A=1,B=0andtarget=0
�
��
�=1.2∗1+0.6*0=1.2
Thisisgreaterthanthethresholdvalueof1,Sotheoutput=1.
�
�=�
�+ƞ(t-o)�
�
�
1=1.2+0.5(0-1)1=0.7
�
2=0.6+0.5(0-1)0=0.6
Now�
�=0.7,�
�=0.6,threshold=1andlearningrateƞ=0.5
7/19/2024 16Dr. Shivashankar, ISE, GAT
A B Y=A+B
(Target)
0 0 0
0 1 0
1 0 0
1 1 1
Problem2:Assume�
1=1.2����
2=0.6,�ℎ���ℎ�??????�=1���??????�����������ƞ=0.5.
ComputeANDgateusingperceptrontrainingrule.
Solution:1.A=0,B=0andtarget=0
�
��
�=�
1�
1+�
2�
2
=1.2*0+0.6*0=0
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0
2.A=0,B=1andtarget=0
�
��
�=1.2∗0+0.6*1=0.6
Thisisnotgreaterthanthethresholdvalueof1,Sotheoutput=0.
3.A=1,B=0andtarget=0
�
��
�=1.2∗1+0.6*0=1.2
Thisisgreaterthanthethresholdvalueof1,Sotheoutput=1.
�
�=�
�+ƞ(t-o)�
�
�
1=1.2+0.5(0-1)1=0.7
�
2=0.6+0.5(0-1)0=0.6
Now�
�=0.7,�
�=0.6,threshold=1andlearningrateƞ=0.5
7/19/2024 16Dr. Shivashankar, ISE, GAT
A B Y=A+B
(Target)
0 0 0
0 1 0
1 0 0
1 1 1
Problems
For�
�=0.7,�
�=0.6,threshold=1andlearningrateƞ=0.5
1.A=0,B=0andtarget=0
�
��
�=�
1�
1+�
2�
2
=0.7*0+0.6*0=0
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0
2.A=0,B=1andtarget=0
�
��
�=0.7∗0+0.6*1=0.6
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0.
3.A=1,B=0andtarget=0
�
��
�=0.7∗1+0.6*0=0.7
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0.
4.A=1,B=1andtarget=1
�
��
�=0.7∗1+0.6*1=1.3
Thisisgreaterthanthethresholdvalueof1.
Sotheoutput=1.
7/19/2024 17Dr. Shivashankar, ISE, GAT
A
B
0.7
0.6
∈ ??????=1
Weighted sum
Output
For�
�=0.7,�
�=0.6,threshold=1andlearningrateƞ=0.5
1.A=0,B=0andtarget=0
�
��
�=�
1�
1+�
2�
2
=0.7*0+0.6*0=0
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0
2.A=0,B=1andtarget=0
�
��
�=0.7∗0+0.6*1=0.6
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0.
3.A=1,B=0andtarget=0
�
��
�=0.7∗1+0.6*0=0.7
Thisisnotgreaterthanthethresholdvalueof1.
Sotheoutput=0.
4.A=1,B=1andtarget=1
�
��
�=0.7∗1+0.6*1=1.3
Thisisgreaterthanthethresholdvalueof1.
Sotheoutput=1.
7/19/2024 17Dr. Shivashankar, ISE, GAT
A
B
0.7
0.6
∈ ??????=1
Weighted sum
Output
Problem
•Problem3:considerX-ORgate,computePerceptrontrainingrulewiththreshold=1and
learningrate=1.5.
•Solution:y=�
1ҧ�
2+ҧ�
1�
2
•Y=�
1+�
2
•Where,�
1=�
1ҧ�
2(��������1),
•�
2=ҧ�
1�
2(Function2)
•Y=�
1OR�
2(Function3)
•Firstfunction:�
1=�
1ҧ�
2
•Assumetheinitialweightsare�
11=�
21=1
•Threshold=1andLearningrate=1.5
7/19/2024 18Dr. Shivashankar, ISE, GAT
�
1 �
2 y
0 0 0
0 1 1
1 0 1
1 1 0
�
1
�
2
�
1
Y
�
2
�
1 �
11
�
2
�
12
�
21
�
22
�
1
�
2
y
�
1 �
2 �
1
0 0 0
0 1 0
1 0 1
1 1 0
•Problem3:considerX-ORgate,computePerceptrontrainingrulewiththreshold=1and
learningrate=1.5.
•Solution:y=�
1ҧ�
2+ҧ�
1�
2
•Y=�
1+�
2
•Where,�
1=�
1ҧ�
2(��������1),
•�
2=ҧ�
1�
2(Function2)
•Y=�
1OR�
2(Function3)
•Firstfunction:�
1=�
1ҧ�
2
•Assumetheinitialweightsare�
11=�
21=1
•Threshold=1andLearningrate=1.5
7/19/2024 18Dr. Shivashankar, ISE, GAT
�
1 �
2 y
0 0 0
0 1 1
1 0 1
1 1 0
�
1
�
2
�
1
Y
�
2
�
1 �
11
�
2
�
12
�
21
�
22
�
1
�
2
y
�
1 �
2 �
1
0 0 0
0 1 0
1 0 1
1 1 0
Problem
(0,0)�
1��=�
�,�∗�
�=1∗0+1∗0=0(output=0)
(0,1)�
1��=1∗0+1∗1=1(output=1)
�
�,�=�
�,�+ƞ(t-o)�
�
�
11=1+1.5(0-1)0=1
�
21=1+1.5(0-1)1=-0.5
Now,�
11=1,�
21=-0.5,threshold=1andlearningrate=1.5
(0,0)�
1��=�
�,�∗�
�=1∗0+(−0.5)∗0=0(output=0)
(0,1)�
1��=�
�,�∗�
�=1∗0+−0.5∗1=−0.5(output=0)
(1,0)�
1��=�
�,�∗�
�=1∗1+(−0.5)∗0=1(output=1)
(1,1)�
1��=�
�,�∗�
�=1∗1+(−0.5)∗1=0.5(output=0)
……………………………………………………………………………………………………………………………………
Secondfunction:�
2=ҧ�
1�
2
•Assumetheinitialweightsare�
12=�
22=1
•Threshold=1andLearningrate=1.5
•(0,0)�
2��=�
�,�∗�
�=1∗0+1∗0=0(output=0)
•(0,1)�
2��=1∗0+1∗1=1(output=1)
•(1,0)�
2��=1∗1+1∗0=1(output=1)
7/19/2024 19Dr. Shivashankar, ISE, GAT
�
1 �
2 �
2
0 0 0
0 1 1
1 0 0
0 0 0
(0,0)�
1��=�
�,�∗�
�=1∗0+1∗0=0(output=0)
(0,1)�
1��=1∗0+1∗1=1(output=1)
�
�,�=�
�,�+ƞ(t-o)�
�
�
11=1+1.5(0-1)0=1
�
21=1+1.5(0-1)1=-0.5
Now,�
11=1,�
21=-0.5,threshold=1andlearningrate=1.5
(0,0)�
1��=�
�,�∗�
�=1∗0+(−0.5)∗0=0(output=0)
(0,1)�
1��=�
�,�∗�
�=1∗0+−0.5∗1=−0.5(output=0)
(1,0)�
1��=�
�,�∗�
�=1∗1+(−0.5)∗0=1(output=1)
(1,1)�
1��=�
�,�∗�
�=1∗1+(−0.5)∗1=0.5(output=0)
……………………………………………………………………………………………………………………………………
Secondfunction:�
2=ҧ�
1�
2
•Assumetheinitialweightsare�
12=�
22=1
•Threshold=1andLearningrate=1.5
•(0,0)�
2��=�
�,�∗�
�=1∗0+1∗0=0(output=0)
•(0,1)�
2��=1∗0+1∗1=1(output=1)
•(1,0)�
2��=1∗1+1∗0=1(output=1)
7/19/2024 19Dr. Shivashankar, ISE, GAT
�
1 �
2 �
2
0 0 0
0 1 1
1 0 0
0 0 0
Problem
�
�,�=�
�,�+ƞ(t-o)�
�
�
12=1+1.5(0-1)1=-0.5
�
22=1+1.5(0-1)0=1
Now,�
12=-0.5,�
22=1,threshold=1,learningrate=1.5
•(0,0)�
2��=�
�,�∗�
�=−0.5∗0+1∗0=0(output=0)
•(0,1)�
2��=(-0.5)∗0+1∗1=1(output=1)
•(1,0)�
2��=−0.5∗1+1∗0=−0.5(output=0)
•(1,1)�
2��=−0.5∗1+1∗1=0.5(output=0)
•Y=�
1OR�
2�
��=�
1�
1+�
2�
2
•Assumetheinitialweightsare XORtable
•�
1=�
2=1,threshold=1,learningrate=1.5
•(0,0)�
��=�
�∗�
�=1∗0+1∗0=0(output=0)
•(0,1)�
��=1∗0+1∗1=1(output=1)
•(1,0)�
��=1∗1+1∗0=1(output=1)
•(0,0)�
��=1∗0+1∗0=0(output=0)
•∴�
11=1,�
12=−0.5,�
21=−0.5,�
22=1
•�
1=�
2=1.
7/19/2024 20Dr. Shivashankar, ISE, GAT
�
1�
2�
1�
2�
��
0 0 0 0 0
0 1 0 1 1
1 0 1 0 1
1 1 0 0 0
�
�,�=�
�,�+ƞ(t-o)�
�
�
12=1+1.5(0-1)1=-0.5
�
22=1+1.5(0-1)0=1
Now,�
12=-0.5,�
22=1,threshold=1,learningrate=1.5
•(0,0)�
2��=�
�,�∗�
�=−0.5∗0+1∗0=0(output=0)
•(0,1)�
2��=(-0.5)∗0+1∗1=1(output=1)
•(1,0)�
2��=−0.5∗1+1∗0=−0.5(output=0)
•(1,1)�
2��=−0.5∗1+1∗1=0.5(output=0)
•Y=�
1OR�
2�
��=�
1�
1+�
2�
2
•Assumetheinitialweightsare XORtable
•�
1=�
2=1,threshold=1,learningrate=1.5
•(0,0)�
��=�
�∗�
�=1∗0+1∗0=0(output=0)
•(0,1)�
��=1∗0+1∗1=1(output=1)
•(1,0)�
��=1∗1+1∗0=1(output=1)
•(0,0)�
��=1∗0+1∗0=0(output=0)
•∴�
11=1,�
12=−0.5,�
21=−0.5,�
22=1
•�
1=�
2=1.
7/19/2024 20Dr. Shivashankar, ISE, GAT
�
1�
2�
1�
2�
��
0 0 0 0 0
0 1 0 1 1
1 0 1 0 1
1 1 0 0 0
Problem
•Problem3:ConsiderNANDgate,computePerceptrontrainingrulewithW1=1.2,
W2=0.6threshold=-1andlearningrate=1.5.
•Solution:
7/19/2024 21Dr. Shivashankar, ISE, GAT
A B Y=�.�
0 0 1
0 1 1
1 0 1
1 1 0
•Problem3:ConsiderNANDgate,computePerceptrontrainingrulewithW1=1.2,
W2=0.6threshold=-1andlearningrate=1.5.
•Solution:
7/19/2024 21Dr. Shivashankar, ISE, GAT
A B Y=�.�
0 0 1
0 1 1
1 0 1
1 1 0
Problem
•Problem3:ConsiderNORgate,computePerceptrontrainingrulewithW1=0.6,W1=1.
threshold=-0.5andlearningrate=1.5.
•Solution:
7/19/2024 22Dr. Shivashankar, ISE, GAT
A B Y=�+�
0 0 1
0 1 0
1 0 0
1 1 0
•Problem3:ConsiderNORgate,computePerceptrontrainingrulewithW1=0.6,W1=1.
threshold=-0.5andlearningrate=1.5.
•Solution:
7/19/2024 22Dr. Shivashankar, ISE, GAT
A B Y=�+�
0 0 1
0 1 0
1 0 0
1 1 0
Problem
•Problem6:ComputeANDgateusingsingleperceptrontrainingrule.
•Solution:
A
B
Y=ቊ
1����+�>0
0����+�≤0
•Assumew1=1,w2=1andbias=-1
•Perceptrontrainingrule:y=w1x1+w2x2+b
•Ifx1=0,x2=0,then0+0-1=-1
0 1 0+1-1=0 -1
1 0 1+0-1=0 1 y=1
1 1 1+1-1=1
1
7/19/2024 23Dr. Shivashankar, ISE, GAT
A B Y=A.B
0 0 0
0 1 0
1 0 0
1 1 1
Y=A.B
AND
b
x1
x2
•Problem6:ComputeANDgateusingsingleperceptrontrainingrule.
•Solution:
A
B
Y=ቊ
1����+�>0
0����+�≤0
•Assumew1=1,w2=1andbias=-1
•Perceptrontrainingrule:y=w1x1+w2x2+b
•Ifx1=0,x2=0,then0+0-1=-1
0 1 0+1-1=0 -1
1 0 1+0-1=0 1 y=1
1 1 1+1-1=1
1
7/19/2024 23Dr. Shivashankar, ISE, GAT
A B Y=A.B
0 0 0
0 1 0
1 0 0
1 1 1
Y=A.B
AND
b
x1
x2
Problems
•Problem7:ComputeORgateusingsingleperceptrontrainingrule.
•Solution:
Y=ቊ
1����+�>0
0����+�≤0
•Assumew1=1,w2=1andbias=-1
•Perceptrontrainingrule:y=w1x1+w2x2+b
•Ifx1=0,x2=0,then0+0-1=-1
0 1 0+1-1=0
Butoutputy=0andtarget=1,misclassification,letuschangethew1=1,w2=2.
Then,y=w1x1+w2x2+bandw1=1,w2=2,b=-1
For(0,0),y=0+0-1=-1
(0,1),y=1x0+2x1-1=1
(1,0),y=1x1+2x0-1=0,Butoutput=0andtarget=1,
misclassification,soletuschangethew1=2andw2=2
(0,0),y=0+0-1=-1
(0,1),y=2x0+2x1-1=1
(1,0),y=2x1+0x2-1=1
(1,1),y=2x1+2x1-1=3
7/19/2024 24Dr. Shivashankar, ISE, GAT
A B Y=A+B
0 0 0
0 1 1
1 0 1
1 1 1
OR
b
x1
x2
2
2
Y=1
-1
•Problem7:ComputeORgateusingsingleperceptrontrainingrule.
•Solution:
Y=ቊ
1����+�>0
0����+�≤0
•Assumew1=1,w2=1andbias=-1
•Perceptrontrainingrule:y=w1x1+w2x2+b
•Ifx1=0,x2=0,then0+0-1=-1
0 1 0+1-1=0
Butoutputy=0andtarget=1,misclassification,letuschangethew1=1,w2=2.
Then,y=w1x1+w2x2+bandw1=1,w2=2,b=-1
For(0,0),y=0+0-1=-1
(0,1),y=1x0+2x1-1=1
(1,0),y=1x1+2x0-1=0,Butoutput=0andtarget=1,
misclassification,soletuschangethew1=2andw2=2
(0,0),y=0+0-1=-1
(0,1),y=2x0+2x1-1=1
(1,0),y=2x1+0x2-1=1
(1,1),y=2x1+2x1-1=3
7/19/2024 24Dr. Shivashankar, ISE, GAT
A B Y=A+B
0 0 0
0 1 1
1 0 1
1 1 1
OR
b
x1
x2
2
2
Y=1
-1
Gradient Descent and the Delta Rule
•Itisalsoimportantbecausegradientdescentcanserveasthebasisforlearningalgorithmsthat
mustsearchthroughhypothesisspaces.
•Thedeltatrainingruleisbestunderstoodbyconsideringthetaskoftraininganunthresholded
perceptron;thatis,alinearunitforwhichtheoutputoisgivenby
o=�
0+�
1�
1+�
2�
2+…………..+�
��
�
O(Ԧ�)=�.Ԧ�
Thus, a linear unit corresponds to the first stage of a perceptron, without the threshold.
Althoughtherearemanywaystodefinethiserror,onecommonmeasurethatwillturnouttobe
especiallyconvenientis
��=
1
2
σ
����
�−�
�
2
whereDisthesetoftrainingexamples,�
�isthetargetoutputfortrainingexampled,and�
�isthe
outputofthelinearunitfortrainingexampled.
GradientDescentandtheDeltaRuleforeachweightchangedby
∆�
��=ƞ??????
�??????
�
??????
�=??????
�(1=??????
�)(�
�−??????
�)ifjisanoutputunit
??????
�=??????
�(1=??????
�)σ
�??????
��
��ifjisahiddenunit
Whereƞisaconstantcalledthelearningrate
�
�isthecorrectteacheroutputforunitj
??????
�istheerrormeasureforunitj
7/19/2024 25Dr. Shivashankar, ISE, GAT
•Itisalsoimportantbecausegradientdescentcanserveasthebasisforlearningalgorithmsthat
mustsearchthroughhypothesisspaces.
•Thedeltatrainingruleisbestunderstoodbyconsideringthetaskoftraininganunthresholded
perceptron;thatis,alinearunitforwhichtheoutputoisgivenby
o=�
0+�
1�
1+�
2�
2+…………..+�
��
�
O(Ԧ�)=�.Ԧ�
Thus, a linear unit corresponds to the first stage of a perceptron, without the threshold.
Althoughtherearemanywaystodefinethiserror,onecommonmeasurethatwillturnouttobe
especiallyconvenientis
��=
1
2
σ
����
�−�
�
2
whereDisthesetoftrainingexamples,�
�isthetargetoutputfortrainingexampled,and�
�isthe
outputofthelinearunitfortrainingexampled.
GradientDescentandtheDeltaRuleforeachweightchangedby
∆�
��=ƞ??????
�??????
�
??????
�=??????
�(1=??????
�)(�
�−??????
�)ifjisanoutputunit
??????
�=??????
�(1=??????
�)σ
�??????
��
��ifjisahiddenunit
Whereƞisaconstantcalledthelearningrate
�
�isthecorrectteacheroutputforunitj
??????
�istheerrormeasureforunitj
7/19/2024 25Dr. Shivashankar, ISE, GAT
The Backpropagation Algorithm
•Backpropagationisaneffectivealgorithmusedtotrainartificialneuralnetworks,especiallyin
feed-forwardneuralnetworks.
•Itsaniterativealgorithm,thathelpstominimizethecostfunctionbydeterminingwhichweights
andbiasesshouldbeadjustedtominimizethelossbymovingdowntowardsthegradientofthe
error.
LetusConsidernetworkswithmultipleoutputunitsratherthansingleunitsasbefore,webeginby
redefiningEtosumtheerrorsoverallofthenetworkoutputunits
��=
1
2
���
���������
�
��−�
��
2
Where,outputsisthesetofoutputunitsinthenetwork,and�
��and�
��arethetargetandoutput
valuesassociatedwiththek
th
outputunitandtrainingexampled.
7/19/2024 26Dr. Shivashankar, ISE, GAT
•Backpropagationisaneffectivealgorithmusedtotrainartificialneuralnetworks,especiallyin
feed-forwardneuralnetworks.
•Itsaniterativealgorithm,thathelpstominimizethecostfunctionbydeterminingwhichweights
andbiasesshouldbeadjustedtominimizethelossbymovingdowntowardsthegradientofthe
error.
LetusConsidernetworkswithmultipleoutputunitsratherthansingleunitsasbefore,webeginby
redefiningEtosumtheerrorsoverallofthenetworkoutputunits
��=
1
2
���
���������
�
��−�
��
2
Where,outputsisthesetofoutputunitsinthenetwork,and�
��and�
��arethetargetandoutput
valuesassociatedwiththek
th
outputunitandtrainingexampled.
7/19/2024 26Dr. Shivashankar, ISE, GAT
Case1:Computeandderivetheincrement(∆)foroutputunitweightinThe
BackpropagationAlgorithm(�
�)
Derivation:
��
??????
����
??????
=
��
??????�0
??????
�0
??????����
??????
��
??????
����
??????
=
�
��
??????
1
2
σ
����������
�−�
�
2
=
�
��
??????
1
2
�
�−�
�
2
=
1
2
*2(�
�−�
�)
�(�
??????−�
??????)
��
??????
=-(�
�−�
�)
��
??????
����
??????
=-(�
�−�
�)�
�(1−�
�)=-�
�(1−�
�)(�
�−�
�)
And
??????
�≠−
��
??????
����
??????
=�
�(1−�
�)(�
�−�
�) .
∆�
��=ƞ??????
��
��=ƞ�
�(1−�
�)(�
�−�
�)�
��. �
�
7/19/2024 27Dr. Shivashankar, ISE, GAT
Type
equation
here.
∈ +
����??????
BackpropagationAlgorithm(�
�)
Derivation:
��
??????
����
??????
=
��
??????�0
??????
�0
??????����
??????
��
??????
����
??????
=
�
��
??????
1
2
σ
����������
�−�
�
2
=
�
��
??????
1
2
�
�−�
�
2
=
1
2
*2(�
�−�
�)
�(�
??????−�
??????)
��
??????
=-(�
�−�
�)
��
??????
����
??????
=-(�
�−�
�)�
�(1−�
�)=-�
�(1−�
�)(�
�−�
�)
And
??????
�≠−
��
??????
����
??????
=�
�(1−�
�)(�
�−�
�) .
∆�
��=ƞ??????
��
��=ƞ�
�(1−�
�)(�
�−�
�)�
��. �
�
7/19/2024 27Dr. Shivashankar, ISE, GAT
Type
equation
here.
∈ +
����??????
Case2:Computeandderivetheincrement(∆)forhiddenunitweightinThe
BackpropagationAlgorithm(�
�)
Derivation:
7/19/2024 28Dr. Shivashankar, ISE, GAT
BackpropagationAlgorithm(�
�)
Derivation:
7/19/2024 28Dr. Shivashankar, ISE, GAT
The Backpropagation Algorithm
BACKPROPOGATION(training_examples,ƞ,�
��,�
���,�
������)
Eachtrainingexampleisapairoftheform(Ԧ�,Ԧ�),whereԦ�isthevectorofnetworkinputvalues,
and�isthevectoroftargetnetworkoutputvalues.
ƞisthelearningrate(e.g.,.O5).�
��,isthenumberofnetworkinputs,�
ℎ�����thenumberof
unitsinthehiddenlayer,and�
���,thenumberofoutputunits.
Theinputfromunitiintounitjisdenoted�
�,�,andtheweightfromunititounitjis
denoted�
�,�.
Createafeed-forwardnetworkwith�
��inputs,�
ℎ�����hiddenunits,and�
���outputunits.
Untiltheterminationconditionismet,do
Foreach(Ԧ�,Ԧ�)intraining_examples,do
Propagatetheinputforwardthroughthenetwork:
1,InputtheinstanceԦ�tothenetworkandcomputetheoutput�
�of
everyunituinthenetwork:�
�=σ
��
��∗�
�����
�=��
�=
1
1+�
−??????
??????
Propagatetheerrorsbackwardthroughthenetwork:
2.Foreachnetworkoutputunitk,calculateitserrorterm??????
�.
??????
�←�
�1−�
��
�−�
�
3.Foreachhiddenunith,calculateitserrorterm??????
ℎ.
??????
�←�
ℎ1−�
ℎ
���������
�
�ℎ??????
�
4.Updateeachnetworkweight�
��
�
��←�
��+∆�
��
where,∆�
��=ƞ??????
��
��
7/19/2024 29Dr. Shivashankar, ISE, GAT
BACKPROPOGATION(training_examples,ƞ,�
��,�
���,�
������)
Eachtrainingexampleisapairoftheform(Ԧ�,Ԧ�),whereԦ�isthevectorofnetworkinputvalues,
and�isthevectoroftargetnetworkoutputvalues.
ƞisthelearningrate(e.g.,.O5).�
��,isthenumberofnetworkinputs,�
ℎ�����thenumberof
unitsinthehiddenlayer,and�
���,thenumberofoutputunits.
Theinputfromunitiintounitjisdenoted�
�,�,andtheweightfromunititounitjis
denoted�
�,�.
Createafeed-forwardnetworkwith�
��inputs,�
ℎ�����hiddenunits,and�
���outputunits.
Untiltheterminationconditionismet,do
Foreach(Ԧ�,Ԧ�)intraining_examples,do
Propagatetheinputforwardthroughthenetwork:
1,InputtheinstanceԦ�tothenetworkandcomputetheoutput�
�of
everyunituinthenetwork:�
�=σ
��
��∗�
�����
�=��
�=
1
1+�
−??????
??????
Propagatetheerrorsbackwardthroughthenetwork:
2.Foreachnetworkoutputunitk,calculateitserrorterm??????
�.
??????
�←�
�1−�
��
�−�
�
3.Foreachhiddenunith,calculateitserrorterm??????
ℎ.
??????
�←�
ℎ1−�
ℎ
���������
�
�ℎ??????
�
4.Updateeachnetworkweight�
��
�
��←�
��+∆�
��
where,∆�
��=ƞ??????
��
��
7/19/2024 29Dr. Shivashankar, ISE, GAT
Problems
Problem1:Assumethattheneuronshaveasigmoidactivationfunction,performaforwardpassand
backwardpassonthenetwork.Assumethattheactualoutputofyis0.5andlearningrateis1.
Performanotherforwardpass.
Solution:Forwardpass:computeoutputfor�
3,�
4and�
5
�
�=
�
�
��∗�
� �
�=��
�=
1
1+�
−??????
??????
�
1=�
13∗�
1+�
23∗�
2=0.1*0.35+0.8*0.9=0.755
�
3=�(�
1)=
1
1+�
−0.755
=0.68
�
2=�
14∗�
1+�
24∗�
2=0.4*0.35+0.6*0.9=0.68
�
4=�(�
2)=
1
1+�
−0.68
=0.6637
�
3=�
35∗�
3+�
45∗�
4=0.3*0.68+0.9*0.667=0.801
7/19/2024 30Dr. Shivashankar, ISE, GAT
�
1=0.35
�
2=0.9
??????
3
??????
4
??????
5
�
13=0.1
�
14=0.4
�
23=0.8
�
24=0.6
�
45=0.9
�
35=0.3
�
5
Output y
�
3
�
4
Problem1:Assumethattheneuronshaveasigmoidactivationfunction,performaforwardpassand
backwardpassonthenetwork.Assumethattheactualoutputofyis0.5andlearningrateis1.
Performanotherforwardpass.
Solution:Forwardpass:computeoutputfor�
3,�
4and�
5
�
�=
�
�
��∗�
� �
�=��
�=
1
1+�
−??????
??????
�
1=�
13∗�
1+�
23∗�
2=0.1*0.35+0.8*0.9=0.755
�
3=�(�
1)=
1
1+�
−0.755
=0.68
�
2=�
14∗�
1+�
24∗�
2=0.4*0.35+0.6*0.9=0.68
�
4=�(�
2)=
1
1+�
−0.68
=0.6637
�
3=�
35∗�
3+�
45∗�
4=0.3*0.68+0.9*0.667=0.801
7/19/2024 30Dr. Shivashankar, ISE, GAT
�
1=0.35
�
2=0.9
??????
3
??????
4
??????
5
�
13=0.1
�
14=0.4
�
23=0.8
�
24=0.6
�
45=0.9
�
35=0.3
�
5
Output y
�
3
�
4
Conti..
�
5=�(�
3)=
1
1+�
−0.801
=0.69(Networkoutput)
∴�����=�
�??????�??????��−�
5=0.5-0.69=-0.19
…………………………………………………………………………………………………………………………………………
Eachweightchangedby
∆�
��=ƞ??????
�??????
�
??????
�=??????
�(1=??????
�)(�
�−??????
�)ifjisanoutputunit
??????
�=??????
�(1=??????
�)σ
�??????
��
��ifjisahiddenunit
Whereƞisaconstantcalledthelearningrate
�
�isthecorrectteacheroutputforunitj
??????
�istheerrormeasureforunitj
Backwardpass:Compute??????
3,??????
4���??????
5
Foroutputunit:
??????
5=�
51−�
5(�
�??????�??????��−�
5)=0.69*(1-0.69)*(0.5-0.69)=-0.0406
Forhiddenunit:
??????
3=�
31−�
3(�
35∗??????
5)=0.68*(1-0.68)*(0.3*-0.0406)=-0.00265
??????
4=�
41−�
4(�
45∗??????
5)=0.6637*(1-0.6637)*(0.9*-0.0406)=-0.0082
7/19/2024 31Dr. Shivashankar, ISE, GAT
�
5=�(�
3)=
1
1+�
−0.801
=0.69(Networkoutput)
∴�����=�
�??????�??????��−�
5=0.5-0.69=-0.19
…………………………………………………………………………………………………………………………………………
Eachweightchangedby
∆�
��=ƞ??????
�??????
�
??????
�=??????
�(1=??????
�)(�
�−??????
�)ifjisanoutputunit
??????
�=??????
�(1=??????
�)σ
�??????
��
��ifjisahiddenunit
Whereƞisaconstantcalledthelearningrate
�
�isthecorrectteacheroutputforunitj
??????
�istheerrormeasureforunitj
Backwardpass:Compute??????
3,??????
4���??????
5
Foroutputunit:
??????
5=�
51−�
5(�
�??????�??????��−�
5)=0.69*(1-0.69)*(0.5-0.69)=-0.0406
Forhiddenunit:
??????
3=�
31−�
3(�
35∗??????
5)=0.68*(1-0.68)*(0.3*-0.0406)=-0.00265
??????
4=�
41−�
4(�
45∗??????
5)=0.6637*(1-0.6637)*(0.9*-0.0406)=-0.0082
7/19/2024 31Dr. Shivashankar, ISE, GAT
Conti..
Computenewweights:
∆�
��=ƞ??????
�??????
�
∆�
45=ƞ??????
5�
4=1*-0.0406*0.6637=-0.0269
�
45(new)=∆�
45+�
45(old)=-0.0269+0.9=0.8731
∆�
14=ƞ??????
4�
1=1*-0.0082*0.35=-0.00287
�
14(���)=∆�
14+�
14(�??????�)=-0.00287+0.4=0.3971
Similarly,updateallotherweights
7/19/2024 32Dr. Shivashankar, ISE, GAT
i J �
��??????
��
�ƞ Updated
�
��
1 3 0.1-0.00265 0.351 0.0991
2 3 0.8-0.00265 0.9 1 0.7976
1 4 0.4-0.0082 0.351 0.3971
2 4 0.6-0.0082 0.9 1 0.5926
3 5 0.3-0.0406 0.681 0.2724
4 5 0.9-0.0406 0.66371 0.8731
Computenewweights:
∆�
��=ƞ??????
�??????
�
∆�
45=ƞ??????
5�
4=1*-0.0406*0.6637=-0.0269
�
45(new)=∆�
45+�
45(old)=-0.0269+0.9=0.8731
∆�
14=ƞ??????
4�
1=1*-0.0082*0.35=-0.00287
�
14(���)=∆�
14+�
14(�??????�)=-0.00287+0.4=0.3971
Similarly,updateallotherweights
7/19/2024 32Dr. Shivashankar, ISE, GAT
i J �
��??????
��
�ƞ Updated
�
��
1 3 0.1-0.00265 0.351 0.0991
2 3 0.8-0.00265 0.9 1 0.7976
1 4 0.4-0.0082 0.351 0.3971
2 4 0.6-0.0082 0.9 1 0.5926
3 5 0.3-0.0406 0.681 0.2724
4 5 0.9-0.0406 0.66371 0.8731
Conti..
Updatednetwork
2
nd
timeForwardpass:Forwardpass:computeoutputfor�
�,�
�and�
�
�
�=
�
�
��∗�
� �
�=��
�=
1
1+�
−??????
??????
�
1=�
13∗�
1+�
23∗�
2=0.0991*0.35+0.7976*0.9=0.7525
�
3=�(�
1)=
1
1+�
−0.7525
=0.6797
�
2=�
14∗�
1+�
24∗�
2=0.3971*0.35+0.5926*0.9=0.6723
�
4=�(�
2)=
1
1+�
−0.6723
=0.6620
�
3=�
35∗�
3+�
45∗�
4=0.2724*0.6797+0.8731*0.6620=0.7631
�
5=�(�
3)=
1
1+�
−0.7631
=0.6820(Networkoutput)
Error=�
�??????�??????��−�
5=0.5−0.6820=-0.182
7/19/2024 33Dr. Shivashankar, ISE, GAT
Updatednetwork
2
nd
timeForwardpass:Forwardpass:computeoutputfor�
�,�
�and�
�
�
�=
�
�
��∗�
� �
�=��
�=
1
1+�
−??????
??????
�
1=�
13∗�
1+�
23∗�
2=0.0991*0.35+0.7976*0.9=0.7525
�
3=�(�
1)=
1
1+�
−0.7525
=0.6797
�
2=�
14∗�
1+�
24∗�
2=0.3971*0.35+0.5926*0.9=0.6723
�
4=�(�
2)=
1
1+�
−0.6723
=0.6620
�
3=�
35∗�
3+�
45∗�
4=0.2724*0.6797+0.8731*0.6620=0.7631
�
5=�(�
3)=
1
1+�
−0.7631
=0.6820(Networkoutput)
Error=�
�??????�??????��−�
5=0.5−0.6820=-0.182
7/19/2024 33Dr. Shivashankar, ISE, GAT
Conti..
Problem2:Assumethattheneuronshaveasigmoidactivationfunction,performa
forwardpassandabackwardpassonthenetwork.Assumethattheactualoutputofyis1
andlearningrateis0.9.Performanotherforwardpass.
Solution:
Forwardpass:Computeoutputfor�
4,�
5and�
6
7/19/2024 34Dr. Shivashankar, ISE, GAT
�
1=1
�
3=1
�
2=0
??????
4
??????
5
??????
6
�
1,5=−0.3
�
1,4=0.2
�
2,5=0.1
�
2,4=0.4
�
3,4=−0.5
�
3,5=0.2
�
4,6=-0.3
�
5,6=-0.2
�����??????�����=1
??????
4=-0.4 or Bias
??????
5=0.2
??????
6=0.1
Problem2:Assumethattheneuronshaveasigmoidactivationfunction,performa
forwardpassandabackwardpassonthenetwork.Assumethattheactualoutputofyis1
andlearningrateis0.9.Performanotherforwardpass.
Solution:
Forwardpass:Computeoutputfor�
4,�
5and�
6
7/19/2024 34Dr. Shivashankar, ISE, GAT
�
1=1
�
3=1
�
2=0
??????
4
??????
5
??????
6
�
1,5=−0.3
�
1,4=0.2
�
2,5=0.1
�
2,4=0.4
�
3,4=−0.5
�
3,5=0.2
�
4,6=-0.3
�
5,6=-0.2
�����??????�����=1
??????
4=-0.4 or Bias
??????
5=0.2
??????
6=0.1
Conti..
�
�=
�
�
��∗�
� �
�=��
�=
1
1+�
−??????
??????
�
4=�
14∗�
1+�
24∗�
2+�
34∗�
3+??????
4(orbias)=(0.2*1)+(0.4*0)+(-0.5*1)+(-0.4)=-0.7
�(??????
4)=�
4=�(�
4)=
1
1+�
0.7
=0.332
�
5=�
15∗�
1+�
25∗�
2+�
35∗�
3+??????
5=(-0.3*1)+(0.1*0)+(0.2*1)+0.2=0.1
�(??????
5)=�
5=�(�
5)=
1
1+�
−0.1
=0.525
�
6=�
46∗??????
4+�
56∗??????
5+??????
6=(-0.3*0.332)+(-0.2*0.525)+0.1=-0.105
�(??????
6)=�
6=�(�
6)=
1
1+�
0.105
=0.474
Error=�
�??????�??????��-�
6=1-0.474=0.526
.................................................................................................................................................
Backwardpass:
Foroutputunit:
??????6
=�
6(1-�
6)(�
�??????�??????��-�
6)=0.474*(1-0.474)=0.1311
Forhiddenunit:
??????5
=�
5(1-�
5)�
56*
??????6
=0.525*(1-0.525)*(-0.2*0.1311)=-0.0065
??????4
=�
4(1-�
4)�
46*
??????6
=0.332*(1-0.332)*(-0.3*0.1331)=-0.0087
7/19/2024 35Dr. Shivashankar, ISE, GAT
�
�=
�
�
��∗�
� �
�=��
�=
1
1+�
−??????
??????
�
4=�
14∗�
1+�
24∗�
2+�
34∗�
3+??????
4(orbias)=(0.2*1)+(0.4*0)+(-0.5*1)+(-0.4)=-0.7
�(??????
4)=�
4=�(�
4)=
1
1+�
0.7
=0.332
�
5=�
15∗�
1+�
25∗�
2+�
35∗�
3+??????
5=(-0.3*1)+(0.1*0)+(0.2*1)+0.2=0.1
�(??????
5)=�
5=�(�
5)=
1
1+�
−0.1
=0.525
�
6=�
46∗??????
4+�
56∗??????
5+??????
6=(-0.3*0.332)+(-0.2*0.525)+0.1=-0.105
�(??????
6)=�
6=�(�
6)=
1
1+�
0.105
=0.474
Error=�
�??????�??????��-�
6=1-0.474=0.526
.................................................................................................................................................
Backwardpass:
Foroutputunit:
??????6
=�
6(1-�
6)(�
�??????�??????��-�
6)=0.474*(1-0.474)=0.1311
Forhiddenunit:
??????5
=�
5(1-�
5)�
56*
??????6
=0.525*(1-0.525)*(-0.2*0.1311)=-0.0065
??????4
=�
4(1-�
4)�
46*
??????6
=0.332*(1-0.332)*(-0.3*0.1331)=-0.0087
7/19/2024 35Dr. Shivashankar, ISE, GAT
Conti..
Computenewweights
∆�
��=ƞ??????
��
�
∆�
46=ƞ??????
6�
4=0.9*0.1311*0.332=0.03917
�
46(new)=∆�
46+�
46(old)=0.03917+(-0.3)=-0.261
∆�
14=ƞ??????
4�
1=0.9*-0.0087*1=-0.0078
�
14(new)=∆�
14+�
14(old)=-0.0076+0.2=0.192
7/19/2024 36Dr. Shivashankar, ISE, GAT
ij�
��??????
��
�ƞUpdated�
��
46-0.30.13110.3320.9 -0.261
56-0.20.13110.5250.9 -0.138
140.2-0.00871 0.9 0.192
15-0.3-0.00651 0.9 -0.306
240.4-0.00870 0.9 0.4
250.1-0.00650 0.9 0.1
34-0.5-0.00871 0.9 -0.508
350.2-0.00651 0.9 0.194
Computenewweights
∆�
��=ƞ??????
��
�
∆�
46=ƞ??????
6�
4=0.9*0.1311*0.332=0.03917
�
46(new)=∆�
46+�
46(old)=0.03917+(-0.3)=-0.261
∆�
14=ƞ??????
4�
1=0.9*-0.0087*1=-0.0078
�
14(new)=∆�
14+�
14(old)=-0.0076+0.2=0.192
7/19/2024 36Dr. Shivashankar, ISE, GAT
ij�
��??????
��
�ƞUpdated�
��
46-0.30.13110.3320.9 -0.261
56-0.20.13110.5250.9 -0.138
140.2-0.00871 0.9 0.192
15-0.3-0.00651 0.9 -0.306
240.4-0.00870 0.9 0.4
250.1-0.00650 0.9 0.1
34-0.5-0.00871 0.9 -0.508
350.2-0.00651 0.9 0.194
Conti..
Updatednetwork:
2
nd
timeForwardpass:Forwardpass:computeoutputfor�
�,�
�and�
�
�
4=�
14∗�
1+�
24∗�
2+�
34∗�
3+??????
4=(0.192*1)+(0.4*0)+(-0.508*1)+(-0.408)=-0.724
�(??????
4)=�
4=�(�
4)=
1
1+�
0.724
=0.327
�
5=�
15∗�
1+�
25∗�
2+�
35∗�
3+??????
5=(-0.306*1)+(0.1*0)+(0.194*1)+(0.194)=0.082
�(??????
5)=�
5=�(�
5)=
1
1+�
−0.082
=0.520
�
6=�
46∗??????
4+�
56∗??????
5+??????
6=(-0.261*0.327)+(-0.138*0.520)+0.218=0.061
�(??????
6)=�
6=�(�
6)=
1
1+�
−0.061.
=0.515(NetworkOutput)
Error=�
�??????�??????��-�
6=1-0.515=0.485
7/19/2024 37Dr. Shivashankar, ISE, GAT
Updatednetwork:
2
nd
timeForwardpass:Forwardpass:computeoutputfor�
�,�
�and�
�
�
4=�
14∗�
1+�
24∗�
2+�
34∗�
3+??????
4=(0.192*1)+(0.4*0)+(-0.508*1)+(-0.408)=-0.724
�(??????
4)=�
4=�(�
4)=
1
1+�
0.724
=0.327
�
5=�
15∗�
1+�
25∗�
2+�
35∗�
3+??????
5=(-0.306*1)+(0.1*0)+(0.194*1)+(0.194)=0.082
�(??????
5)=�
5=�(�
5)=
1
1+�
−0.082
=0.520
�
6=�
46∗??????
4+�
56∗??????
5+??????
6=(-0.261*0.327)+(-0.138*0.520)+0.218=0.061
�(??????
6)=�
6=�(�
6)=
1
1+�
−0.061.
=0.515(NetworkOutput)
Error=�
�??????�??????��-�
6=1-0.515=0.485
7/19/2024 37Dr. Shivashankar, ISE, GAT
Categories
TechnologyDownload
Download Slideshow Get the original presentation fileQuick Actions
Statistics
- Views 1,090
- Slides 37
- Age 501 days
Related Slideshows
11
8-top-ai-courses-for-customer-support-representatives-in-2025.pptx
JeroenErne2
46 views
10
7-essential-ai-courses-for-call-center-supervisors-in-2025.pptx
JeroenErne2
46 views
13
25-essential-ai-courses-for-user-support-specialists-in-2025.pptx
JeroenErne2
37 views
11
8-essential-ai-courses-for-insurance-customer-service-representatives-in-2025.pptx
JeroenErne2
34 views
21
Know for Certain
DaveSinNM
21 views
17
PPT OPD LES 3ertt4t4tqqqe23e3e3rq2qq232.pptx
novasedanayoga46
26 views