Classification by back propagation, multi layered feed forward neural networks -bihira aggrey

Classification by Back Propagation, Multi-layered feed forward Neural Networks - Provides a basic introduction of classification in data mining with neural networks

1. ClassificationbyBackPropagation,Multi-layeredfeed
forwardNeuralNetworks
By
BihiraAggrey

2. NEURAL NETWORKS
•Aneuralnetworkisasetofconnectedinput/outputunitsinwhicheach
connectionhasaweightassociatedwithit.
•Neuralnetworkslearnbyexamplewithoutnecessarilybeingprogrammed.
•ANNincorporatetwoimportantfundamentalcomponentsofbiological
neuralnets.
 Neurons(Nodes)
 Synapses(Weights)

3. NeuralNetworksarecomputingsystemsinspiredbybiologicalneuronsthat
constituteanimalbrains.Thesesystemslearntodotasksbyconsidering
examplesgenerallywithouttaskspecificprogramming.
Neuron vs. Node

4. ARCHITECTURE OF A NEURON MODEL
ConsideranelementaryneuronwithRinputsshownbelow.Eachinputis
weightedwithanappropriatew.Thesumoftheweightedinputsandthe
biasformstheinputtotheactivationfunctionf.

5. NEURAL NETWORK NEURONS -SUMMARIZED!
Receivesn-inputs
Multiplieseachinputbyitsweight
Sumstheweightedinputsandbiasforeachinput
Appliesactivationfunctiontothesumofresults
Outputsresult

6. LEARNING
Duringthelearningphase,thenetworklearnsbyadjustingtheweightssoastobe
abletopredictthecorrectclasslabeloftheinputtuples.Neuralnetworklearningis
alsoreferredtoasconnectionistlearningduetotheconnectionsbetweenunits.

7. Neuralnetworksinvolvelongtrainingtimesandarethereforemoresuitablefor
applicationswherethisisfeasible.
Neuralnetworkshavebeencriticizedfortheirpoorinterpretability.For
example,itisdifficultforhumanstointerpretthesymbolicmeaningbehindthe
learnedweightsandof“hiddenunits”inthenetwork.Thesefeaturesinitially
madeneuralnetworkslessdesirablefordatamining.
Theadvantageisthattheyhowever,includetheirhightoleranceofnoisydata
aswellastheirabilitytoclassifypatternsonwhichtheyhavenotbeentrained.
Theycanbeusedwhenyoumayhavelittleknowledgeoftherelationships
betweenattributesandclasses.

8. A multilayer feed-forward neural
network consists of an input layer,
one or more hidden
layers, and an output layer.
The inputsto the network
correspond to the attributes
measured for each trainingtuple.
The inputs are fed simultaneously into
the units making up the inputlayer.
These inputs pass through the input
layer and are then weighted and fed
simultaneously to a second layer
known as a hiddenlayer.
The outputs of the hidden layer units
can be input to another hidden
layer, and so on.
This structure is called multilayerbecause it has a layer of
processing units (i.e., the hidden units) in addition to the
output units. These networks are called feedforward
because the output from one layer of neurons feeds
forward into the next layer of neurons.
A multilayer feed-forward neural network

9. BACKPROPAGATION
Thisisaneuralnetworklearningalgorithm.
Backpropagationlearnsbyiterativelyprocessingadatasetoftraining
tuples,comparingthenetwork’spredictionforeachtuplewiththeactual
knowntargetvalue.
Thetargetvaluemaybetheknownclasslabelofthetrainingtuple(for
classificationproblems)oracontinuousvalue(fornumericprediction).

10. Foreachtrainingtuple,theweightsaremodifiedsoastominimizethe
mean-squarederrorbetweenthenetwork’spredictionandtheactual
targetvalue.
Thesemodificationsaremadeinthebackwarddirectionthatisfromthe
outputlayerthrougheachhiddenlayerdowntothefirsthiddenlayer
hencethenamebackpropagation.

11. AlgorithmBackpropagation.Neuralnetworklearningfor
classificationornumericprediction,usingthe
backpropagationalgorithm.
•Input D,adatasetconsistingofthetrainingtuplesand
theirassociatedtargetvalues;
l,thelearningrate;
network,amultilayerfeed-forwardnetwork.
•Output Atrainedneuralnetwork.

12. BACKPROPAGATION ALGORITHM -METHOD

14. •Initializetheweights.Theweightsinthenetworkareinitializedtosmall
randomnumbersforexamplerangingfrom-1.0to1.0,or-0.5to0.5.Each
unithasabiasassociatedwithit.
•Eachtrainingtuple,X,isprocessedbythefollowingsteps.

15. PROPAGATE INPUTS FORWARD

16. Backpropagatethe error
Theerrorispropagatedbackwardbyupdatingtheweightsandbiasestoreflect
theerrorofthenetwork’sprediction.
Foraunitjintheoutputlayer,theerrorErrjiscomputedby
Errj=Oj(1-Oj)(Tj-Oj)
whereOjistheactualoutputofunitj,andTjisthe
knowntargetvalueofthegiventrainingtuple.

17. For the error of a hidden layer unit j, the weighted sum of the errors of
the units connected to unit jin the next layer are considered.
The error for the hidden layer unit j is then calculated from;
where Wjkis the weight of the connection from unit j
to a unit k in the next higher layer, and Errkis the
error of unit k.

18. Learningratel,
Thevariablelisthelearningrate,aconstanttypicallyhavingavaluebetween0.0
and1.0.
Backpropagationlearnsusingagradientdescentmethodtosearchforasetofweights
thatfitsthetrainingdatasoastominimizethemeansquareddistancebetweenthe
network’sclasspredictionandtheknowntargetvalueofthetuples.
Thelearningratehelpsavoidgettingstuckatalocalminimumindecisionspace(i.e.,
wheretheweightsappeartoconverge,butarenottheoptimumsolution)and
encouragesfindingtheglobalminimum.
Ifthelearningrateistoosmall,thenlearningwilloccurataveryslowpace.Ifthe
learningrateistoolarge,thenoscillationbetweeninadequatesolutionsmayoccur.A
ruleofthumbistosetthelearningrateto1/t,wheretisthenumberofiterations
throughthetrainingsetsofar.

19. The network was trained with a run consisting of 60,000
iterations, Since the training data has 150 cases, each case was
presented to the network 400 times.
The output values of neurons in the output layer are used to
compute the error. This error is used to adjust the weights of
all the connections in the network using the backward
propagation.
The backprop algorithm cycles through two distinct passes, a forward pass followed by a
backward pass through the layers of the network. The algorithm alternates between these
passes several times as it scans the training data. Typically, the training data has to be
scanned several times before the networks ”learns” to make good classifications.