Neural Networks Basic Concepts and Deep Learning

NN

1. Course
Outcomes
Aftercompletionofthiscourse,studentswillbeableto
Understandmachine-learningconcepts.
UnderstandandimplementClassificationconcepts.
UnderstandandanalysethedifferentRegression
algorithms.
ApplytheconceptofUnsupervisedLearning.
ApplytheconceptsofArtificialNeuralNetworks.

2. Topics
Biological Neurons and Biological
Neural Networks
Artificial Neural Networks
Perceptron
Activation Functions
Applications of Artificial Neural
Networks (ANNs)
Neural Network
Types of Artificial Neural Networks
FeedforwardNeuralNetworks
(FNN)/Multi-LayerPerceptron
(MLP)
ConvolutionalNeuralNetworks
(CNN)
RecurrentNeuralNetworks
(RNN)
TransformerNeuralNetworks
Autoencoders
Generative Adversarial
Networks(GANs)
CompetitiveNeuralNetworks

3. Biological Neurons and
Biological Neural
Networks

4. Biological
Neurons
Thesearethenervecellsinyourbrainandnervoussystem.
Eachneuronhasdendrites(whichreceivesignals),acellbody(which
processesthesignals),andanaxon(whichsendssignalstootherneurons).
Neuronscommunicatewitheachotherthroughelectricalimpulses,
formingcomplexnetworkstohelpusthink,move,feel,andlearn.

5. Biological
Neural
Networks
Abiologicalneuralnetworkisacollectionofinterconnected
neuronsinyourbrain.
Thesenetworksareresponsibleforeverythingyoudo,like
recognizingfaces,rememberingthings,orsolving
problems.
Theneuronsareconnectedbysynapses.Whenyoulearn
somethingnew,theconnectionsbetweencertainneurons
strengthen,whichhelpsyourememberorperformtasks
better.

6. Biological
Neural
Networks

7. Artificial Neural
Networks and
Perceptron

8. ANN
ANNsarecomputersystemsdesignedtomimichow
biologicalneuronswork,butthey’remadeupofmath,
notcells.
Anartificialneurontakesininformation,processesit,
andsendsanoutput,muchlikehowabiologicalneuron
works.
Whenmanyartificialneuronsareconnectedtogether,
theyformanartificialneuralnetwork,whichcanlearn
todothingslikerecognizingobjectsinpictures,
predictingoutcomes,orplayingvideogames.

9. ANN
Theterm"Artificialneuralnetwork"referstoabiologically
inspiredsub-fieldofartificialintelligencemodeledafterthe
brain.
AnArtificialneuralnetworkisusuallyacomputational
networkbasedonbiologicalneuralnetworksthatconstruct
thestructureofthehumanbrain.
Similartoahumanbrainhasneuronsinterconnectedto
eachother,artificialneuralnetworksalsohaveneurons
thatarelinkedtoeachotherinvariouslayersofthe
networks.Theseneuronsareknownasnodes.

10. Thegivenfigureillustratesthetypicaldiagramof
BiologicalNeuralNetwork.
ThetypicalArtificialNeuralNetworklookssomething
likethegivenfigure.

11. Thegivenfigureillustratesthetypicaldiagramof
BiologicalNeuralNetwork.
Biological Neural NetworkArtificial Neural Network
Dendrites Inputs
Cell nucleus Nodes
Synapse Weights
Axon Output
ThetypicalArtificialNeuralNetworklookssomething
likethegivenfigure.

12. Perceptron
A‘Perceptron’isthebasicbuildingblock,orsingle
node,ofaneuralnetworkinspiredfromtheneurons
thatarefoundinthebrain.
Itoperatesbytakinginasetofinputs,calculatinga
weightedsum,addingabiasterm,andthenapplying
anactivationfunctiontothissumtoproducean
output.

14. The inner
working of a
perceptron is
as follows:

15. Perceptron
Learning
Perceptronlearningreferstohowaperceptronadjusts
itsweightstoimproveaccuracy.
Whentheperceptronmakesawrongprediction,it
learnsbychangingtheweightstogetclosertothe
correctanswernexttime.
Overtime,throughrepeatedadjustments,the
perceptronlearnstomakebetterpredictions.

16. Activation Function

17. Activation
Function
Activation:Inbiologicalneurons,activationisthe
firingrateoftheneuronwhichhappenswhenthe
impulsesarestrongenoughtoreachthethreshold.In
artificialneuralnetworks,Amathematicalfunction
knownasanactivationfunctionmapstheinputtothe
output,andexecutesactivations.

19. Activation
Functions
Thepurposeofanactivationfunctionistointroduce
non-linearityintothemodel,allowingthenetworkto
learnandrepresentcomplexpatternsinthedata.
Theactivationfunctiondecideswhetheraneuron
shouldbeactivatedornotbycalculatingtheweighted
sumandfurtheraddingbiastoit.Thepurposeofthe
activationfunctionistointroducenon-linearityintothe
outputofaneuron.

23. Linear
Function
Equation:Linearfunctionhastheequationsimilartoasofastraightline
i.e.y=x
Nomatterhowmanylayerswehave,ifallarelinearinnature,thefinal
activationfunctionoflastlayerisnothingbutjustalinearfunctionofthe
inputoffirstlayer.
Range:-infto+inf
Uses:Linearactivationfunctionisusedatjustoneplacei.e.outputlayer.
Issues:Ifwewilldifferentiatelinearfunctiontobringnon-linearity,result
willnomoredependoninput“x”andfunctionwillbecomeconstant,it
won’tintroduceanyground-breakingbehaviortoouralgorithm.
Forexample:Calculationofpriceofahouseisaregressionproblem.House
pricemayhaveanybig/smallvalue,sowecanapplylinearactivationat
outputlayer.Eveninthiscaseneuralnetmusthaveanynon-linearfunction
athiddenlayers.

25. RELU Function
ItStandsforRectifiedlinearunit.Itisthemostwidelyusedactivation
function.ChieflyimplementedinhiddenlayersofNeuralnetwork.
Equation:-A(x)=max(0,x).Itgivesanoutputxifxispositiveand0
otherwise.
ValueRange:-[0,inf)
Nature:-non-linear,whichmeanswecaneasilybackpropagatethe
errorsandhavemultiplelayersofneuronsbeingactivatedbythe
ReLUfunction.
Uses:-ReLuislesscomputationallyexpensivethantanhandsigmoid
becauseitinvolvessimplermathematicaloperations.Atatimeonlya
fewneuronsareactivatedmakingthenetworksparsemakingit
efficientandeasyforcomputation.
Insimplewords,RELUlearnsmuchfasterthansigmoidandTanh
function.

28. TanhFunction
Theactivationthatworksalmostalwaysbetterthansigmoidfunctionis
TanhfunctionalsoknownasTangentHyperbolicfunction.It’sactually
mathematicallyshiftedversionofthesigmoidfunction.Botharesimilar
andcanbederivedfromeachother.
Equation :-
f(x) = tanh(x) = 2/(1 + e-2x) –1
OR
tanh(x) = 2 * sigmoid(2x) –1
Value Range :--1 to +1
Nature :-non-linear
Uses:-Usuallyusedinhiddenlayersofaneuralnetworkasit’svalues
liesbetween-1to1hencethemeanforthehiddenlayercomesoutbe
0orveryclosetoit,hencehelpsincenteringthedatabybringingmean
closeto0.Thismakeslearningforthenextlayermucheasier.

30. Sigmoid
Function
Itisafunctionwhichisplottedas‘S’shapedgraph.
Equation:A=1/(1+e
-x
)
Nature:Non-linear.NoticethatXvaluesliesbetween-2to
2,Yvaluesareverysteep.Thismeans,smallchangesinx
wouldalsobringaboutlargechangesinthevalueofY.
ValueRange:0to1
Uses:Usuallyusedinoutputlayerofabinaryclassification,
whereresultiseither0or1,asvalueforsigmoidfunctionlies
between0and1onlyso,resultcanbepredictedeasilyto
be1ifvalueisgreaterthan0.5and0otherwise.

32. Softmax
Function
Thesoftmaxfunctionisalsoatypeofsigmoidfunctionbutis
handywhenwearetryingtohandlemulti-class
classificationproblems.
Nature:-non-linear
Uses:-Usuallyusedwhentryingtohandlemultipleclasses.
thesoftmaxfunctionwascommonlyfoundintheoutput
layerofimageclassificationproblems.Thesoftmaxfunction
wouldsqueezetheoutputsforeachclassbetween0and1
andwouldalsodividebythesumoftheoutputs.
Output:-Thesoftmaxfunctionisideallyusedintheoutput
layeroftheclassifierwhereweareactuallytryingtoattain
theprobabilitiestodefinetheclassofeachinput.

34. Applications of Artificial
Neural Networks
(ANNs)

35. Applications of
Artificial
Neural
Networks
(ANNs):
Image Recognition (e.g., face detection,
object classification)
Speech Recognition (e.g., virtual assistants
like Siri and Alexa)
Natural Language Processing (NLP) (e.g.,
language translation, text generation)
Medical Diagnosis (e.g., detecting diseases
from medical images or records)
Financial Predictions (e.g., stock market
forecasting, fraud detection)
Autonomous Vehicles (e.g., self-driving cars,
traffic sign recognition)
Recommender Systems (e.g., Netflix,
Amazon, YouTube recommendations)
Robotics (e.g., robot vision, control systems)
Customer Support Chatbots(e.g.,
automating responses to queries)
Game AI (e.g., AI playing video games or
board games like Go)
Time Series Forecasting (e.g., weather
prediction, sales forecasting)
Anomaly Detection (e.g., cybersecurity,
equipment failure detection)
Art Generation (e.g., creating artwork, music
composition)
Social Media Monitoring (e.g., sentiment
analysis, spam detection)
Personalized Marketing (e.g., targeted
advertising, customer behavior prediction)

36. Neural Network

37. Perceptron is a single layer neural
networkand a multi-layer perceptron is
called Neural Networks.

38. Neural
Network

39. Neural
Network
ThisNeuralNetworkorArtificialNeuralNetworkhas
multiplehiddenlayersthatmakeitamultilayerneural
Networkanditisfeed-forwardbecauseitisanetworkthat
followsatop-downapproachtotrainthenetwork.Inthis
networktherearethefollowinglayers:
InputLayer
HiddenLayer
OutputLayer

41. Neural
Network

42. Neural
Network

43. Neural
Network

44. Neural
Network

45. Neural
Network

46. Neural
Network

47. Neural
Network
Thebasicruleofthumbisifyoureallydon’tknowwhat
activationfunctiontouse,thensimplyuseRELUasitisa
generalactivationfunctioninhiddenlayersandisusedin
mostcasesthesedays.
Ifyouroutputisforbinaryclassificationthen,sigmoid
functionisverynaturalchoiceforoutputlayer.
Ifyouroutputisformulti-classclassificationthen,
Softmaxisveryusefultopredicttheprobabilitiesofeach
classes.

48. Types of Artificial
Neural Networks

49. Types of
Artificial
Neural
Networks
FeedforwardNeuralNetworks(FNN)/Multi-Layer
Perceptron(MLP)
ConvolutionalNeuralNetworks(CNN)
RecurrentNeuralNetworks(RNN)
TransformerNeuralNetworks
Autoencoders
GenerativeAdversarialNetworks(GANs)
CompetitiveNeuralNetworks

50. Feedforward
Neural
Networks
(FNN) / Multi-
Layer
Perceptron
(MLP)
Thesearethesimplestneuralnetworkswhere
informationflowsinonedirection:fromtheinputtothe
output.
Thinkofitlikeafunnel—yougivesomeinputatthetop
(likenumbers),andthenetworkprocessestheinputlayer
bylayeruntilitreachesafinaldecisionattheoutput(like
yes/no,orclassifyinganimage).
Example:Yougiveitanimage,andittellsyouwhetherit’s
acatoradog.

51. Convolutional
Neural
Networks
(CNN)
CNNsarespecialneuralnetworksdesignedforimage
data.
Theyhavelayersthatscantheimagepiecebypieceto
findpatternslikeedges,colors,orshapes,whichhelps
thenetworkunderstandthecontentoftheimage.
Example:CNNsareusedinapplicationslikefacial
recognitionoridentifyingobjectsinphotos.

52. Recurrent
Neural
Networks
(RNN)
RNNsareusedwhenyoudealwithsequencesofdata
(likesentences,timeseries,orspeech).
Thesenetworksrememberwhattheyprocessedearlier
inthesequence,allowingthemtomakedecisions
basedonbothcurrentinputandpastinputs(like
havingashort-termmemory).
Example:RNNscanbeusedtopredictthenextwordin
asentenceorrecognizespokenwordsinspeech.

53. Transformer
Neural
Networks:
Transformersarepowerfulnetworksforprocessing
languageorsequentialdata.
UnlikeRNNs,theylookatthewholesentenceatonce
insteadofonewordatatime,whichmakesthemfaster
andmoreaccurate.
Theyusesomethingcalledself-attentiontofocuson
importantpartsoftheinputsequence.
Example:GPT-3(themodelbehindchatbotslikethis)isa
transformer,usedforgeneratingtext,answering
questions,ortranslatinglanguages.

54. Autoencoders
Autoencodersareusedfortaskslikecompressing
dataorfindingpatterns.
Theytakeinputdata,reduceittoasimplerversion
(compression),andthentrytorebuilditbacktoits
originalform.
Example:Autoencodersareusedtocompressimages
intosmallerfilesorcleannoisydata.

55. Generative
Adversarial
Networks
(GANs)
GANsinvolvetwoneuralnetworksworkingtogether.
Onetriestocreatefakedata(likefakeimages),and
theothertriestodetectwhichdataisrealandwhich
isfake.
Overtime,thegeneratornetworkgetsbetterat
makingrealisticfakedata,whilethediscriminatorgets
betteratspottingfakes.
Example:GANsareusedtocreaterealisticfake
images,likegeneratingpicturesofpeoplewhodon’t
exist.

56. Competitive
Neural
Networks
Incompetitivenetworks,neuronscompetewitheach
other,andonlythemostactiveoneis“activated.”
Thesenetworksareoftenusedinclusteringtasks
wheresimilardatapointsaregroupedtogether.
Example:Self-OrganizingMaps(SOMs)areatypeof
competitivenetworkusedtoclusterdataintogroups,
likefindingsimilarpatternsinlargedatasets.

57. Type of Neural Network Purpose Structure Best For How It Works
Feedforward Neural Networks
(FNN) / Multi-Layer Perceptron
(MLP)
Basic neural network for general
tasks like classification and
regression.
Data moves in one direction,
layer by layer (input →hidden
→output).
Classifying simple data like
numbers or basic images.
Takes input, processes it
through layers, and gives a
prediction (e.g., "cat" or "dog").
Convolutional Neural
Networks (CNN)
Specialized for image and video
data.
Uses layers that look at parts of
an image (like scanning) to
detect patterns (edges, shapes).
Recognizing objects in images,
like face detection or medical
image analysis.
Scans parts of the image to
learn what it contains (e.g.,
looks for edges, colors,
textures).
Recurrent Neural Networks
(RNN)
Processes sequences of data,
where order matters.
Includes loops to remember
past data (short-term memory).
Time-series data, speech
recognition, text prediction
(e.g., what comes next).
Remembers previous data (like
words in a sentence or past
events in time) to make a
decision.
Transformer Neural Networks
Fast and efficient for language
and sequential tasks.
Processes whole sequences at
once (no need for memory
loops).
Natural language processing
(NLP) like translation, text
generation.
Looks at all words in a sentence
together and figures out their
relationships using "self-
attention."
Autoencoders
Compresses data, finds
patterns, or reduces noise.
Has an encoder to shrink the
input and a decoder to
reconstruct it.
Data compression, noise
reduction (like removing
background noise).
Compresses the input into a
smaller form and then tries to
rebuild it to match the original.
Generative Adversarial
Networks (GANs)
Generates new, realistic data
like images or videos.
Has two networks: a generator
(creates fake data) and a
discriminator (detects fake
data).
Creating realistic images,
videos, or even music (like
deepfakes).
The generator makes fake data,
and the discriminator tries to
catch the fakes, making the
generator improve over time.
Competitive Neural Networks
Groups similar data into clusters
without supervision.
Neurons compete to be the
most active; only one "wins" and
is activated.
Clustering similar patterns in
data, like organizing large
datasets into groups.
Neurons compete, and the
"winner" learns from the data,
helping to group similar data
points together.