Google net

Slide share for deep learning google network.

1. Korea University, Department of Computer Science & Radio Communication Engineering 2016010646 Bumsoo Kim G o o g l e N e t Everything about GoogleNet and its architecture Bumsoo Kim Presentation 2017 1

2. Bumsoo Kim Presentation 2017 2 What is ‘ GoogleNet ’? GoogleNet

3. Bumsoo Kim Presentation 2017 3 What is ‘ GoogleNet ’? GoogleNet

4. Bumsoo Kim Presentation 2017 4 What is ‘ GoogleNet ’? GoogleNet

5. Bumsoo Kim Presentation 2017 5 What is ‘ GoogleNet ’? GoogleNet

6. Bumsoo Kim Presentation 2017 6 What is ‘ GoogleNet ’? GoogleNet ILSVRC(ImageNet) 2014 Winning Model Google, “Going Deeper with Convolutions”(CVPR2015) Top5 Error Rate = 6.7% Revolution of Depth (22 layers)

7. Bumsoo Kim Presentation 2017 7 How deep is ‘ GoogleNet ’? Revolution of Depth

8. Bumsoo Kim Presentation 2017 8 Deeper is better? Revolution of Depth The winning models of 2014 was distinguishable by its depth !

9. Bumsoo Kim Presentation 2017 9 Deeper is better? Revolution of Depth The winning models of 2014 was distinguishable by its depth ! So, does deeper network is the key to better performance?

10. Bumsoo Kim Presentation 2017 10 Deeper is better? Revolution of Depth The winning models of 2014 was distinguishable by its depth ! So, does deeper network is the key to better performance? The answer is…

11. Bumsoo Kim Presentation 2017 11 Deeper is better? Revolution of Depth The winning models of 2014 was distinguishable by its depth ! So, does deeper network is the key to better performance? The answer is… NO!

12. Bumsoo Kim Presentation 2017 12 Deeper is NOT better Obstacles of Depth Overfitting Unbalance & Aliasing

13. Bumsoo Kim Presentation 2017 13 Deeper is NOT better Obstacles of Depth Overfitting Unbalance & Aliasing

14. Bumsoo Kim Presentation 2017 14 Deeper is NOT better Obstacles of Depth Overfitting Unbalance & Aliasing

15. Bumsoo Kim Presentation 2017 15 Deeper is NOT better Obstacles of Depth Overfitting Unbalance & Aliasing

16. Bumsoo Kim Presentation 2017 16 Deeper is NOT better Obstacles of Depth Overfitting Unbalance & Aliasing

17. Bumsoo Kim Presentation 2017 17 Deeper is NOT better Obstacles of Depth Overfitting Unbalance & Aliasing

18. Bumsoo Kim Presentation 2017 18 ZF-Net did not provide a fundamental solution Obstacles of Depth ZF-net fundamentally failed in increasing depth

19. Bumsoo Kim Presentation 2017 19 ZF-Net did not provide a fundamental solution Obstacles of Depth ZF-net fundamentally failed in increasing depth 2014 top models succeeded in addressing this problem

20. Bumsoo Kim Presentation 2017 20 ZF-Net did not provide a fundamental solution Obstacles of Depth ZF-net fundamentally failed in increasing depth 2014 top models succeeded in addressing this problem HOW?

21. Bumsoo Kim Presentation 2017 21 Unique module of Googlenet Inception

22. Bumsoo Kim Presentation 2017 22 Network in Network Inception nodules

23. Bumsoo Kim Presentation 2017 23 Network in Network Inception nodules NIN (Network In Network) – Min Lin, 2013, Network in Network local receptive linear features local receptive linear+non-linear features non-linear features

24. Bumsoo Kim Presentation 2017 24 Network in Network Inception nodules NIN (Network In Network) – Min Lin, 2013, Network in Network non-linear features 90% of computation

25. Bumsoo Kim Presentation 2017 25 Network in Network Inception nodules NIN (Network In Network) – Min Lin, 2013, Network in Network Inception Average Pooling

26. Bumsoo Kim Presentation 2017 26 1x1 Convolutions Inception nodules

27. Bumsoo Kim Presentation 2017 27 1x1 Convolutions Inception nodules Original model

28. Bumsoo Kim Presentation 2017 28 1x1 Convolutions Inception nodules Original model Various filters for various feature scales

29. Bumsoo Kim Presentation 2017 29 1x1 Convolutions Inception nodules Original model Expensive unit : Larger filter sizes are very expensive!!

30. Bumsoo Kim Presentation 2017 30 1x1 Convolutions Inception nodules Original model 1x1 Convolution

31. Bumsoo Kim Presentation 2017 31 1x1 Convolutions Inception nodules 1x1 Convolution Dimension reduction Hebbian Principle Neurons that fire together, wire together 동일 차원에서 activate 된 neuron 들은 비슷한 성질을 가진 것들로 묶일 수 있다 . c2 (large) c3 (small)

32. Bumsoo Kim Presentation 2017 32 Do we really need 5x5 filters? Inception Factorizing Convolutions 5*5 = 25 → 2*(3*3) = 18 => 28% decrease of parameters!

33. Bumsoo Kim Presentation 2017 33 Do we really need 5x5 filters? Inception Factorizing Convolutions 5*5 = 25 → 2*(3*3) = 18 => 28% decrease of parameters!

34. Bumsoo Kim Presentation 2017 34 How Effective is it? Inception 130M 4.5B

35. Bumsoo Kim Presentation 2017 35 Why do we need it? Auxiliary Classifier Gradient Vanishing Problem ReLU doesn’t provide a fundamental solution Overlapping multiple ReLUs cause the same problem!

36. Bumsoo Kim Presentation 2017 36 What is wrong about it? Gradient Vanishing Vanishing Gradients on Sigmoids * Deep Networks use activation function for non-linearity. *“Saturated”: Weight ‘W’ is too large. => ‘ z’is 0 or 1 => z*(1-z) = 0 z = 1/(1 + np.exp (- np.dot ( W , x ))) # forward pass dx = np.dot (W.T, z *(1-z)) # backward pass : local gradient for x dW = np.outer ( z *(1-z), x ) # backward pass : local gradient for W * Local gradient is small : 0 <= z*(1-z) <= 0.25 (when z=0.5) : top layers are not trained

37. Bumsoo Kim Presentation 2017 37 Details are not explained in the paper Auxiliary Classifier Training Deeper Convolutional Networks with Deep SuperVision (2014, Liwei Chang, Chen-Yu Lee)

38. Bumsoo Kim Presentation 2017 38 Before Gradient Vanishes ! Auxiliary Classifier Training Deeper Convolutional Networks with Deep SuperVision (2014, Liwei Chang, Chen-Yu Lee) 깊이가 깊어질수록 vanishing 될 우려가 높아지므로 , 얕은 깊이에서도 업데이트 !

39. Bumsoo Kim Presentation 2017 39 Empirical Decision Auxiliary Classifier Training Deeper Convolutional Networks with Deep SuperVision (2014, Liwei Chang, Chen-Yu Lee) 넣는 위치는 이론적 근거 없이 순수하게 “ 경험적 ” 으로 판단한다 .

40. Bumsoo Kim Presentation 2017 40 How Effective is it? Auxiliary Classifier Training Deeper Convolutional Networks with Deep SuperVision (2014, Liwei Chang, Chen-Yu Lee)

41. Bumsoo Kim Presentation 2017 41 Train it yourself ! GoogleNet

42. Bumsoo Kim Presentation 2017 42 Train it yourself ! GoogleNet

43. Bumsoo Kim Presentation 2017 43 Train it yourself ! GoogleNet

44. Bumsoo Kim Presentation 2017 44 Train it yourself ! GoogleNet

45. Bumsoo Kim Presentation 2017 45 Train it yourself ! GoogleNet

46. Bumsoo Kim Presentation 2017 46 Train it yourself ! GoogleNet

47. Bumsoo Kim Presentation 2017 47 Train it yourself ! GoogleNet

48. Bumsoo Kim Presentation 2017 48 Train it yourself ! GoogleNet

49. Bumsoo Kim Presentation 2017 49 Train it yourself ! GoogleNet

50. Bumsoo Kim Presentation 2017 50 Train it yourself ! GoogleNet

51. Bumsoo Kim Presentation 2017 51 Train it yourself ! GoogleNet

52. Bumsoo Kim Presentation 2017 52 Train it yourself ! GoogleNet

53. Bumsoo Kim Presentation 2017 53 Train it yourself ! GoogleNet

54. Bumsoo Kim Presentation 2017 Train it yourself ! GoogleNet

55. Bumsoo Kim Presentation 2017 55 Train it yourself ! GoogleNet

56. Bumsoo Kim Presentation 2017 56 Train it yourself ! GoogleNet

57. T h a n k Y o u Bumsoo Kim Presentation 2017 57