How neural networks work. Demonstration by Brandon Rohrer

Machine Learning, Neural Networks

1. How neural networks work Brandon Rohrer e2eml.school

2. A four pixel camera e2eml.school

3. Categorize images solid vertical diagonal horizontal e2eml.school

4. Categorize images solid vertical diagonal horizontal e2eml.school

5. Categorize images solid vertical diagonal horizontal e2eml.school

6. Categorize images solid vertical diagonal horizontal e2eml.school

7. solid vertical diagonal horizontal e2eml.school

8. Simple rules can’t do it solid vertical diagonal horizontal e2eml.school

9. Input neurons e2eml.school

10. Pixel brightness -1.0 -.75 -.50 -.25 0.0 +.25 +.50 +.75 +1.0 e2eml.school

11. .75 -.75 0.0 .50 Input vector e2eml.school

12. Receptive fields e2eml.school

13. A neuron + e2eml.school

14. Sum all the inputs + .75 -.75 0.0 .50 .50 .50 0.00 -.75 .75 .50 e2eml.school

15. Weights + .75 -.75 0.0 .50 .50 1.0 1.0 1.0 1.0 .50 0.00 -.75 .75 .50 x 1.0 x 1.0 x 1.0 x 1.0 e2eml.school

16. Weights + .75 -.75 0.0 .50 -1.075 -.2 0.0 .8 -.5 .50 0.00 -.75 .75 -1.075 -.2 0.0 .8 -.5 x x x x e2eml.school

17. Weights + .75 -.75 0.0 .50 -1.075 -.2 0.0 .8 -.5 .50 0.00 -.75 .75 -1.075 -.2 0.0 .8 -.5 x x x x e2eml.school

18. Squash the result + .75 -.75 0.0 .50 -1.075 -.746 e2eml.school

19. Hyperbolic tangent (tanh) squashing function 1.0 .5 -1.0 -.5 1.0 .5 1.5 2.0 -1.0 -.5 -1.5 -2.0 e2eml.school

20. tanh squashing function 1.0 .5 -1.0 -.5 1.0 .5 1.5 2.0 -1.0 -.5 -1.5 -2.0 Your number goes in here e2eml.school

21. tanh squashing function 1.0 .5 -1.0 -.5 1.0 .5 1.5 2.0 -1.0 -.5 -1.5 -2.0 e2eml.school

22. tanh squashing function 1.0 .5 -1.0 -.5 1.0 .5 1.5 2.0 -1.0 -.5 -1.5 -2.0 The squashed version comes out here e2eml.school

23. tanh squashing function 1.0 .5 -1.0 -.5 1.0 .5 1.5 2.0 -1.0 -.5 -1.5 -2.0 e2eml.school

24. tanh squashing function 1.0 .5 -1.0 -.5 1.0 .5 1.5 2.0 -1.0 -.5 -1.5 -2.0 e2eml.school

25. No matter what you start with, the answer stays between -1 and 1. 1.0 .5 -1.0 -.5 1.0 .5 1.5 2.0 -1.0 -.5 -1.5 -2.0 e2eml.school

26. Squash the result + .75 -.75 0.0 .50 -1.075 -.746 e2eml.school

27. Weighted sum-and-squash neuron .75 -.75 0.0 .50 -.746 e2eml.school

28. Make lots of neurons, identical except for weights To keep our picture clear, weights will either be 1.0 (white) -1.0 (black) or 0.0 (missing) e2eml.school

29. Receptive fields get more complex e2eml.school

30. Repeat for additional layers e2eml.school

31. Receptive fields get still more complex e2eml.school

32. Repeat with a variation e2eml.school

33. Rectified linear units (ReLUs) 1.0 .5 -1.0 -.5 1.0 .5 1.5 2.0 -1.0 -.5 -1.5 -2.0 If your number is positive, keep it. Otherwise you get a zero. e2eml.school

34. e2eml.school

35. Add an output layer solid vertical diagonal horizontal e2eml.school

36. solid vertical diagonal horizontal e2eml.school

37. solid vertical diagonal horizontal e2eml.school

38. solid vertical diagonal horizontal e2eml.school

39. solid vertical diagonal horizontal e2eml.school

40. solid vertical diagonal horizontal e2eml.school

41. solid vertical diagonal horizontal e2eml.school

42. solid vertical diagonal horizontal e2eml.school

43. Errors solid vertical diagonal horizontal 1. 0. 0. 0. truth e2eml.school

44. Errors solid vertical diagonal horizontal -.75 -.25 .75 .5 1. 0. 0. 0. truth answer e2eml.school

45. Errors solid vertical diagonal horizontal -.75 -.25 .75 .5 1. 0. 0. 0. truth answer error 1.75 .25 .75 .5 e2eml.school

46. Errors solid vertical diagonal horizontal -.75 -.25 .75 .5 1. 0. 0. 0. truth answer error 1.75 .25 .75 .5 total 3.25 e2eml.school

47. Learn all the weights: Gradient descent Error at: Weight original weight e2eml.school

48. Learn all the weights: Gradient descent Error at: Weight original weight lower weight e2eml.school

49. Learn all the weights: Gradient descent Error at: Weight original weight lower weight higher weight e2eml.school

50. Numerically calculating the gradient is very expensive e2eml.school

51. Calculate the gradient (slope) directly Error at: Weight original weight e2eml.school

52. Slope Error at: Weight original weight change in weight = +1 e2eml.school

53. Slope Error at: Weight original weight change in weight = +1 move along the curve e2eml.school

54. Slope Error at: Weight original weight change in weight = +1 change in error = -2 e2eml.school

55. Slope Error at: Weight original weight change in weight = +1 change in error = -2 slope = change in error change in weight e2eml.school

56. Slope Error at: Weight original weight change in weight = +1 change in error = -2 slope = change in error change in weight = ∆ error ∆ weight = d(error) d(weight) = ∂e ∂w = -2 +1 e2eml.school

57. Slope Error at: Weight original weight You have to know your error function. For example: error = weight ^2 -1 +1 e2eml.school

58. Slope Error at: Weight original weight You have to know your error function. For example: error = weight ^2 ∂e = 2 * weight ∂w -1 +1 e2eml.school

59. Slope Error at: Weight original weight You have to know your error function. For example: error = weight ^2 ∂e = 2 * weight ∂w = 2 * -1 = -2 -1 +1 e2eml.school

60. y = x * w 1 Chaining + + x (input) e (output) y (intermediate value) w 1 w 2 e2eml.school

61. y = x * w 1 ∂y = x ∂w 1 Chaining + + x (input) e (output) y (intermediate value) w 1 w 2 e2eml.school

62. y = x * w 1 ∂y = x ∂w 1 e = y * w 2 ∂e = w 2 ∂y Chaining + + x (input) e (output) y (intermediate value) w 1 w 2 e2eml.school

63. y = x * w 1 ∂y = x ∂w 1 e = y * w 2 ∂e = w 2 ∂y e = x * w 1 * w 2 ∂e = x * w 2 ∂w 1 Chaining + + x (input) e (output) y (intermediate value) w 1 w 2 e2eml.school

64. y = x * w 1 ∂y = x ∂w 1 e = y * w 2 ∂e = w 2 ∂y e = x * w 1 * w 2 ∂e = x * w 2 ∂w 1 ∂e = ∂y * ∂e ∂w 1 ∂w 1 ∂y Chaining + + x (input) e (output) y (intermediate value) w 1 w 2 e2eml.school

65. y = x * w 1 ∂y = x ∂w 1 e = y * w 2 ∂e = w 2 ∂y e = x * w 1 * w 2 ∂e = x * w 2 ∂w 1 ∂e = ∂y * ∂e ∂w 1 ∂w 1 ∂y Chaining + + x (input) e (output) y (intermediate value) w 1 w 2 e2eml.school

66. ∂err = ∂a * ∂b * ∂c * ∂d * … * ∂y * ∂z * ∂err ∂weight ∂weight ∂a ∂b ∂c ∂x ∂y ∂z Chaining + err a weight b c x y z ... e2eml.school

67. Backpropagation + err a weight b c x y z ... e2eml.school

68. ∂err = ∂a * ∂b * ∂c * ∂d * … * ∂y * ∂z * ∂err ∂weight ∂weight ∂a ∂b ∂c ∂x ∂y ∂z Backpropagation + err a weight b c x y z ... e2eml.school

69. ∂err = ∂a * ∂b * ∂c * ∂d * … * ∂y * ∂z * ∂err ∂weight ∂weight ∂a ∂b ∂c ∂x ∂y ∂z Backpropagation + err a weight b c x y z ... e2eml.school

70. ∂err = ∂a * ∂b * ∂c * ∂d * … * ∂y * ∂z * ∂err ∂weight ∂weight ∂a ∂b ∂c ∂x ∂y ∂z Backpropagation + err a weight b c x y z ... e2eml.school

71. ∂err = ∂a * ∂b * ∂c * ∂d * … * ∂y * ∂z * ∂err ∂weight ∂weight ∂a ∂b ∂c ∂x ∂y ∂z Backpropagation + err a weight b c x y z ... e2eml.school

72. Backpropagation + err a weight b c x y z ... e2eml.school