Module1 (2).pptxvgybhunjimko,l.vgbyhnjmk;

bnm

1. CSEN3011 ARTIFICIAL NEURAL NETWORKS O. V. Ramana Murthy

2. Course Outcome 1 2 Understand the origin, ideological basics, Learning process and various Neural Architectures of ANN.

3. Contents 3 Artificial Neuron Neural Network Training with Back propagation Practical issues Common Neural Architectures

4. Reference 4 Chapter 1, 2 Charu C. Aggarwal, “Neural Networks and Deep Learning”, Springer International Publishing AG Deep Learning- Charu Aggarwal https://www.youtube.com/playlist?list=PLLo1RD8Vbbb_6gCyqxG_qzCLOj9EKubw7

5. Neuron 5

6. Perceptron 6 The simplest neural network is referred to as the perceptron. This neural network contains a single input layer and an output node.

7. Artificial Neuron 7   w 2 x n x 1 w n w 1  1 f act   b y x 2

8. Numerical Example 1 8 Calculate the output assuming Binary step activation function 0.6 0.2 0.7 0.3 1 f act   0.45 y= 0.93  

9. Artificial Neuron model 9 Sigmoidal function =1 is Binary sigmoidal function   x 2 x 1 w 2 w 1 y

10. ACTIVATION FUNCTIONS 10 Identity/linear (B) Bipolar Binary step (C) Bipolar sigmoidal step   Source [2]

11. Numerical Example 2 11 Calculate the output assuming binary sigmoidal activation function 0.6 0.2 0.7 0.3 1 f act   0.45 y= 0.93 =1 is Binary sigmoidal function  

12. Pre- and Post-Activation Values 12

13. Linear Separability 13

14. Linearly Separable – AND gate 14 x 1 (input) x 2 (input) Y (output) 1 1 1 1 1

15. Linearly Separable – AND gate 15 Two input sources => Two input neurons One output => One output Neuron. Activation function is binary sigmoidal Derivative  

16. Linearly Separable – AND gate 16 Y   x 2 x 1 1 w 2 w 1 w  f (.)

17. Back-propagation training/algorithm 17 Given: Input vector i th instant , Target . Initialize weights w , w 1 , w 2 and learning rate with some random values in the range [0 1] Output Activation function sigmoidal activation function Compute error: Backpropagate the error to crossing activation function where is the derivative of activation function selected. for sigmoidal activation function  

18. Back-propagation training/algorithm 18 Compute change in weights and bias , , Update the changes in weights and bias Keep repeating the steps 1 – 6, for all input combinations ( 4 nos ). This is one epoch. Run multiple Epochs till the error decreases and stabilizes.  

19. Matrix Notation – AND gate 19 Forward pass After activation function Loss function  

20. Matrix Notation – AND gate 20 Backpropagation Update Weights: Update using This iterative process continues until convergence  

21. (4 Rules)Backpropagating Error 21 Y    f (.) x i    f (.) w i 1. Output Neuron   2. Across Link         3. Weights Update  

22. w 2 (4 Rules)Backpropagating Error 22 x i Type equation here.    f (.) w 1 4. Across Link ( >1 hidden layer )          f (.)     w n    f (.)  

23. The power of nonlinear activation functions in transforming a data set to linear separability 23

24. Linearly not Separable – XOR gate 24 x 1 (input) x 2 (input) Y (output) 1 1 1 1 1 1

25. Linearly not Separable – XOR gate 25 Two input sources => Two input neurons One output => One output Neuron. One hidden layer => 2 neurons Activation function is binary sigmoidal Derivative  

26. Linearly not Separable – XOR gate 26

27. w 2 w 1 v 21 v 12 27 x 2 x 1 v 11 Y 1 1 v 22 v 01 v 02 Z 2 Z 1 w Input layer Hidden layer Output layer

28. Back-propagation Training 28 Given: Inputs , target . Initialize weights and learning rate with some random values Hidden unit , j = 1 to p hidden neurons output , sigmoidal activation function Output unit Output sigmoidal activation function   Feed-forward Phase

29. Back-propagation Training 29 Compute error correction term where is derivative Compute change in weights and bias , send to previous layer Hidden unit Calculate error term Compute change in weights and bias ,   Back-propagation of error Phase

30. Back-propagation Training 30 Each output unit, k = 1 to m update weights and bias Each hidden unit, j = 1 to p update weights and bias Check for stopping criterion e.g. certain number of epochs or when targets are equal/close to network outputs   Weights and Bias update phase

31. w 2 w 1 v 21 v 12 31 x 2 x 1 v 11 Y 1 1 v 22 v 01 v 02 Z 2 Z 1 w     Hidden neuron input computation

32. w 2 w 1 v 21 v 12 32 x 2 x 1 v 11 Y 1 1 v 22 v 01 v 02 Z 2 Z 1 w         Hidden neuron output computation

33. w 2 w 1 v 21 v 12 33 x 2 x 1 v 11 Y 1 1 v 22 v 01 v 02 Z 2 Z 1 w       Output neuron input computation

34. w 2 w 1 v 21 v 12 34 x 2 x 1 v 11 Y 1 1 v 22 v 01 v 02 Z 2 Z 1 w       Output neuron Output computation  

35. w 2 w 1 v 21 v 12 35 x 2 x 1 v 11 Y 1 1 v 22 v 01 v 02 Z 2 Z 1 w       Output Error correction computation  

36. w 2 w 1 v 21 v 12 36 x 2 x 1 v 11 Y 1 1 v 22 v 01 v 02 Z 2 Z 1 w       Output neuron changes updates computation        

37. w 2 w 1 v 21 v 12 37 x 2 x 1 v 11 Y 1 1 v 22 v 01 v 02 Z 2 Z 1 w     Hidden neuron error propagation computation        

38. w 2 w 1 v 21 v 12 38 x 2 x 1 v 11 Y 1 1 v 22 v 01 v 02 Z 2 Z 1 w     Hidden neuron error correction computation            

39. w 2 w 1 v 21 v 12 39 x 2 x 1 v 11 Y 1 1 v 22 v 01 v 02 Z 2 Z 1 w     Hidden neuron changes updates computation           1         2   2    

40. Matrix Notation – XOR gate 40 Given  

41. Matrix Notation – XOR gate 41 Forward pass Hidden layer After activation function  

42. Matrix Notation – XOR gate 42 Forward pass Output layer After activation function Loss function  

43. Matrix Notation – XOR gate 43 Backpropagation Update Weights: Hidden layer gradient  

44. Matrix Notation – XOR gate 44 Update using This iterative process continues until convergence  

45. NN with Two Hidden Layers (HW) 45  

46. Practical Issues – Softmax Layer 46

47. Example 47 z =[2.0 1.0 0.1] Then Let Targets be T = [ 1 0 0] Define Loss function  

48. Example 48 Homework. Loss function is cross entropy  

49. Regularization (to avoid Overfitting) 49 One of the primary causes of corruption of the generalization process is overfitting. The objective is to determine a curve that defines the border of the two groups using the training data.

50. Overfitting 50 One of the primary causes of corruption of the generalization process is overfitting. The objective is to determine a curve that defines the border of the two groups using the training data.

51. Overfitting 51 Some outliers penetrate the area of the other group and disturb the boundary. As Machine Learning considers all the data, even the noise, it ends up producing an improper model (a curve in this case). This would be penny-wise and pound-foolish.

52. Remedy : Regularization 52 Regularization is a numerical method that attempts to construct a model structure as simple as possible. The simplified model can avoid the effects of overfitting at the small cost of performance. Cost function Sum of squared errors  

53. Remedy : Regularization 53 For this reason, overfitting of the neural network can be improved by adding the sum of weights to the cost function, (new) Cost function In order to drop the value of the cost function, both the error and weight should be controlled to be as small as possible. However, if a weight becomes small enough, the associated nodes will be practically disconnected. As a result, unnecessary connections are eliminated, and the neural network becomes simpler.  

54. Add L1 Regularization to XOR Network 54 New Loss function The gradient of the regularized loss w.r.t a weight w is: Update rule for weights w is  

55. Add L2 Regularization to XOR Network 55 New Loss function The gradient of the regularized loss w.r.t a weight w is: Update rule for weights w is  

56. XOR implementation with L1 56 # Apply L2 regularization to weights hidden_layer_weights += learning_rate * (np.dot( hidden_layer_output.T , output_layer_delta ) – sign( hidden_layer_weights ) ) input_layer_weights += learning_rate * (np.dot( inputs.T , hidden_layer_delta ) - sign( input_layer_weights ) ) # Update biases (no regularization applied to biases) hidden_layer_bias += np.sum ( output_layer_delta , axis=0, keepdims =True) * learning_rate input_layer_bias += np.sum ( hidden_layer_delta , axis=0, keepdims =True) * learning_rate

57. XOR implementation with L2 57 # Apply L2 regularization to weights hidden_layer_weights += learning_rate * (np.dot( hidden_layer_output.T , output_layer_delta ) - hidden_layer_weights ) input_layer_weights += learning_rate * (np.dot( inputs.T , hidden_layer_delta ) - input_layer_weights ) # Update biases (no regularization applied to biases) hidden_layer_bias += np.sum ( output_layer_delta , axis=0, keepdims =True) * learning_rate input_layer_bias += np.sum ( hidden_layer_delta , axis=0, keepdims =True) * learning_rate

58. Matrix Notation – XOR gate with L2 Regularization 58 Expanding from Slide 44. Only weights will be updated as follows. Bias values won’t change. This iterative process continues until convergence. L2 regularization penalizes large weights, resulting in slightly smaller weight updates compared to the non-regularized case.  

59. Common Neural Architectures 59 Autoencoder (Module 2) Deep Neural Network (Module 3) Attractor Neural Networks (Module 4) Self-organizing Maps (module 5)

60. Autoencoder 60

61. Autoencoder 61

62. Deep Neural Network 62

63. Attractor Neural Networks 63

64. Self-Organizing Maps 64

65. 65 Thank You All Very Much

66. Appendix: Example Implementation 66 Using Back-propagation network, find the new weights for the network shown aside. Input = [0 1] and target output is 1. use learning rate 0.25 and binary sigmoidal activation function

67. 1. Consolidate the information 67 Given: Inputs [0 1], target 1 . [ [ ]=[0.4 0.1 0.2] Learning rate Activation function is binary sigmoidal Derivative  

68. 2. Feed-forward Phase 68 Hidden unit , j = 1,2 Output , sigmoidal activation function Output unit Output sigmoidal activation function  

69. 2. Feed-forward Phase 69 Hidden unit , j = 1,2 Output , sigmoidal activation function , Output unit Output sigmoidal activation function  

70. 3. Back-propagation of error Phase 70 Compute error correction term Compute change in weights and bias , , ,  

71. 3. Back-propagation of error Phase 71 Compute error correction term Compute change in weights and bias , , , Hidden unit  

72. 3. Back-propagation of error Phase 72 Compute error correction term Compute change in weights and bias , , , Hidden unit  

73. 3. Back-propagation of error Phase 73 Calculate error term  

74. 3. Back-propagation of error Phase 74 Calculate error term  

75. 3. Back-propagation of error Phase 75 Calculate error term Compute change in weights and bias ,  

76. 3. Back-propagation of error Phase 76 Calculate error term Compute change in weights and bias , 0.0118 0.0118  

77. 3. Back-propagation of error Phase 77 Calculate error term Compute change in weights and bias , 0.0118 0.0118 0.0 0.00245  

78. 3. Back-propagation of error Phase 78 Calculate error term Compute change in weights and bias , 0.0118 0.0118 0.0 0.00245  

79. 4. Weights and Bias update phase 79 Each output unit, k = 1 to m update weights and bias ,  

80. 4. Weights and Bias update phase 80 Each output unit, k = 1 to m update weights and bias ,  

81. 4. Weights and Bias update phase 81 Each hidden unit, j = 1 to p update weights and bias  

82. 4. Weights and Bias update phase 82 Each hidden unit, j = 1 to p update weights and bias  

83. 4. Weights and Bias update phase 83 Each hidden unit, j = 1 to p update weights and bias  

84. 84 Epoch v 11 v 21 v 01 v 12 v 22 v 02 0.6 -0.1 0.3 -0.3 0.4 0.5 1 0.6 -0.097 0.303 -0.3 0.401 0.501 Write a program for this case and cross-verify your answers. After how many epochs will the output converge? Epoch z 1 z 2 w 1 w 2 w y 0.549 0.711 0.4 0.1 -0.2 0.523 1 0.5513 0.7113 0.416 0.121 -0.17 0.5363