Perceptron & Neural Networks

Perceptron & Neural Networks

1. Perceptron and Neural Networks Shaik Nagur Shareef Dept. of CSE Vignan’s University Output Values Input Signals (External Stimuli)

2. Contents Introduction Neural Networks Perceptron and Examples Types of NN Applications January 20, 2019 2 Shaik Nagur Shareef

3. Human Information processing system Highly complex, nonlinear, and parallel computer. Has the capability to organize its structural constituents, known as neurons , to perform certain computations. Pattern recognition, Perception, and Motor control. Many times faster than the fastest digital computer in existence today Neurons are information-processing units in the human brain. January 20, 2019 3 Shaik Nagur Shareef Interconnected Neurons forms a neural (nerve) net

4. Human Nervous System Viewed as a Three-stage system. Central to the system is the brain , represented by the neural (nerve) net , which continually receives information, perceives it, and makes appropriate decisions. The arrows pointing from pointing from left to right indicate the forward transmission. The arrows pointing from right to left signify the presence of feedback in the system. The receptors convert stimuli from the human body or the external environment into electrical impulses that convey information to the neural net. The effectors convert electrical impulses generated by the neural net into discernible responses as system outputs. January 20, 2019 Shaik Nagur Shareef 4

5. Machine Information Processing System Neural Networks are made of Artificial Neurons, A Neural Network is a machine that is designed to model the way in which the brain performs a particular task or function of interest. The network is usually implemented by using electronic components or is simulated in software on a digital computer. Neural Networks perform useful computations through a process of learning. January 20, 2019 Shaik Nagur Shareef 5

6. Biological Neuron vs Machine Neuron January 20, 2019 Shaik Nagur Shareef 6

7. Neural Network A Neural Network is a massively parallel distributed processor made up of simple processing units that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: Knowledge is acquired by the network from its environment through a learning process. Interneuron connection strengths, known as synaptic weights, are used to store the acquired knowledge. The procedure used to perform the learning process is called a learning algorithm , the function of which is to modify the synaptic weights of the network in an orderly fashion to attain a desired design objective. The modification of synaptic weights provides the traditional method for the design of neural networks. January 20, 2019 Shaik Nagur Shareef 7

8. Neural Network January 20, 2019 Shaik Nagur Shareef 8

9. Artificial Neuron - [McPi43] In 1943, Warren McCulloch and Walter Pitts introduced one of the first artificial neurons [McPi43]. The main feature of their neuron model is that a weighted sum of input signals is compared to a threshold to determine the neuron output. When the sum is greater than or equal to the threshold, the output is 1. When the sum is less than the threshold, the output is 0. Networks of these neurons in principle, compute any arithmetic or logical function. The parameters of [McPi43] networks had to be designed, as no training method was available. January 20, 2019 Shaik Nagur Shareef 9

10. Perceptron - [Rose58] In the late 1950s, Frank Rosenblatt and several other researchers developed a class of neural networks called perceptrons. The neurons in these networks were similar to those of McCulloch and Pitts. Rosenblatt's key contribution was the introduction of a learning rule for training perceptron networks to solve pattern recognition problems [Rose58]. He proved that his learning rule will always converge to the correct network weights, if weights exist that solve the problem. Learning was simple and automatic. Examples of proper behavior were presented to the network, which learned from its mistakes. The perceptron could even learn when initialized with random values for its weights and biases. January 20, 2019 Shaik Nagur Shareef 10

11. Learning Rule It is a Procedure for modifying the weights and biases of a network. This procedure may also be referred to as a training algorithm. The purpose of the learning rule is to train the network to perform some task. There are many types of neural network learning rules. They fall into three broad categories: Supervised learning Unsupervised learning Reinforcement (or graded) learning January 20, 2019 Shaik Nagur Shareef 11

12. Supervised Learning In supervised learning , the learning rule is provided with a set of examples (the training set ) of proper network behavior: Here p q is an input to the network and t q is the corresponding correct ( target ) output. As the inputs are applied to the network, the network outputs are compared to the targets. The learning rule is then used to adjust the weights and biases of the network in order to move the network outputs closer to the targets. The perceptron learning rule falls in this supervised learning category. January 20, 2019 Shaik Nagur Shareef 12

13. Unsupervised Learning In unsupervised learning , the weights and biases are modified in response to network inputs only. There are no target outputs available. At first glance this might seem to be impractical. How can you train a network if you don't know what it is supposed to do? Most of these algorithms perform some kind of clustering operation. They learn to categorize the input patterns into a finite number of classes. This is especially useful in such applications as vector quantization. January 20, 2019 Shaik Nagur Shareef 13

14. Reinforcement Learning Reinforcement learning is similar to supervised learning, except that, instead of being provided with the correct output for each network input, the algorithm is only given a grade. The grade (or score) is a measure of the network performance over some sequence of inputs. This type of learning is currently much less common than supervised learning. It appears to be most suited to control system applications January 20, 2019 Shaik Nagur Shareef 14

15. Perceptron Architecture January 20, 2019 Shaik Nagur Shareef 15

16. Procedure consider the network weight matrix January 20, 2019 Shaik Nagur Shareef 16

17. Activation Function(hardlim Function) January 20, 2019 Shaik Nagur Shareef 17 Step Function Sigmoid Function

18. Single-Neuron Perceptron -Example Assign Values to weights and bias January 20, 2019 Shaik Nagur Shareef 18

19. Single-Neuron Perceptron -Example Therefore, the network output will be 1 for the region above and to the right of the decision boundary . January 20, 2019 Shaik Nagur Shareef 19

20. Single-Neuron Perceptron -Example January 20, 2019 Shaik Nagur Shareef 20

21. Logic Function: AND gate January 20, 2019 Shaik Nagur Shareef 21

22. Weight Update Rule January 20, 2019 Shaik Nagur Shareef 22

23. Weight Update Rule January 20, 2019 Shaik Nagur Shareef 23

24. Training Algorithms Adjust neural network weights to map inputs to outputs. Use a set of sample patterns where the desired output (given the inputs presented) is known. The purpose is to learn to generalize Recognize features which are common to good and bad exemplars January 20, 2019 Shaik Nagur Shareef 24

25. Training: Key Terms Epoch : Presentation of the entire training set to the neural network. In the case of the AND function an epoch consists of four sets of inputs being presented to the network (i.e. [0,0], [0,1], [1,0], [1,1]) Error : The error value is the amount by which the value output by the network differs from the target value. For example, if we required the network to output 0 and it output a 1, then Error = -1 Online training: Update weights after each sample Offline (batch training): Compute error over all samples Then update weights Training: Backpropagation procedure Gradient descent strategy (usual problems) Prediction: Compute outputs based on input vector & weights January 20, 2019 Shaik Nagur Shareef 25

26. Gradient Descent Concept Error: Sum of squares error of inputs with current weights Compute rate of change of error w.r.t each weight Which weights have greatest effect on error? Effectively, partial derivatives of error w.r.t weights In turn, depend on other weights => chain rule E = G(w) Error as function of weights Find rate of change of error Follow steepest rate of change Change weights so that error is minimized January 20, 2019 Shaik Nagur Shareef 26

27. Gradient Descent Algorithm Gradient-Descent( training_examples , ) Each training example is a pair of the form <(x 1 ,…x n ),t> where (x 1 ,…,x n ) is the vector of input values, and t is the target output value,  is the learning rate (e.g. 0.1) Initialize each w i to some small random value Until the termination condition is met, Do Initialize each w i to zero For each <(x 1 ,…x n ),t> in training_examples Do Input the instance (x 1 ,…,x n ) to the linear unit and compute the output o For each linear unit weight w i Do w i = w i +  (t-o) x i For each linear unit weight wi Do w i =w i +w i January 20, 2019 Shaik Nagur Shareef 27

28. Gradient Descent Algorithm January 20, 2019 Shaik Nagur Shareef 28 (w 1 ,w 2 ) (w 1 + w 1 ,w 2 + w 2 )

29. Back-Propagation A training procedure which allows multi-layer feedforward Neural Networks to be trained; Can theoretically perform “any” input-output mapping; Can learn to solve linearly inseparable problems. For feed-forward networks: A continuous function can be differentiated allowing gradient-descent. Back-propagation is an example of a gradient-descent technique. January 20, 2019 Shaik Nagur Shareef 29

30. Back-Propagation Algorithm Initialize each w i to some small random value Until the termination condition is met, Do For each training example <(x 1 ,…x n ),t> Do Input the instance (x 1 ,…,x n ) to the network and compute the network outputs o k For each output unit k  k =o k (1-o k )(t k -o k ) For each hidden unit h  h =o h (1-o h )  k w h,k  k For each network weight w ,j Do w i,j =w i,j +w i,j where w i,j =   j x i,j January 20, 2019 Shaik Nagur Shareef 30

31. Types of Neural Networks January 20, 2019 Shaik Nagur Shareef 31 Standard NN Recurrent NN Convolutional NN

32. Benefits of Neural Networks Nonlinearity: An artificial neuron can be linear or nonlinear. Input–Output Mapping: A popular paradigm of learning, called learning with a teacher , or supervised learning , involves modification of the synaptic weights of a neural network by applying a set of labelled training examples , or task examples. Adaptivity: Neural networks have a built-in capability to adapt their synaptic weights to changes in the surrounding environment. Evidential Response: In the context of pattern classification, a neural network can be designed to provide information not only about which particular pattern to select , but also about the confidence in the decision made. Fault Tolerance: A neural network, implemented in hardware form, has the potential to be inherently fault tolerant , or capable of robust computation, in the sense that its performance degrades gracefully under adverse operating conditions. January 20, 2019 Shaik Nagur Shareef 32

33. Applications Natural language Processing Optical Character Recognition Speech recognition Neural Machine Translation Video Classification Emotion Recognition Face Recognition Object Detection Image Classification January 20, 2019 Shaik Nagur Shareef 33

34. Thank You Any Questions..?