ML Visuals.pptx

machine learning PPT Template

1. ML Visuals By dair.ai https://github.com/dair-ai/ml-visuals

2. Basic ML Visuals

3. Softmax Convolve Sharpen

4. Softmax Convolve Sharpen

5. Positional Encoding Masked Multi-Head Attention Add & Norm Output Embedding Multi-Head Attention Add & Norm Outputs(shifted right) Positional Encoding Multi-Head Attention Add & Norm Input Embedding Feed Forward Add & Norm Inputs Feed Forward Add & Norm Linear Softmax

6. Multi-Head Attention Add & Norm Input Embedding Output Embedding Feed Forward Add & Norm Masked Multi-Head Attention Add & Norm Multi-Head Attention Add & Norm Feed Forward Add & Norm Linear Softmax Inputs Outputs (shifted right) Positional Encoding Positional Encoding

7. Tokenize I love coding and writing “I love coding and writing”

8. Input Layer Hidden Layers Output Layer X = A [0] a [4] A [1] A [3] X Ŷ a [1] n a [2] 1 a [2] 2 a [2] 3 a [2] n a [3] 1 a [3] 2 a [3] 3 a [3] n A [2] A [4]

9. Input Layer Hidden Layers Output Layer X = A [0] a [4] A [1] A [3] X Ŷ a [1] 1 a [1] 2 a [1] 3 a [1] n a [2] 1 a [2] 2 a [2] 3 a [2] n a [3] 1 a [3] 2 a [3] 3 a [3] n A [2] A [4]

10. Input Layer Hidden Layers Output Layer X = A [0] a [4] A [1] A [3] X Ŷ [1 a ] 1 a [1] 2 a [1] 3 a [1] n a [2] 1 a [2] 2 a [2] 3 a [2] n a [3] 1 a [3] 2 a [3] 3 a [3] n A [2] A [4]

11. NxNx3 +b1 +b2 MxM MxM +b1 +b2 ReLU ReLU a [l] MxMX2 a [l-1] CONV operation

12. NxNx3 +b1 +b2 MxM MxM +b1 +b2 ReLU ReLU MxMX2 CONV operation

13. NxNx3 +b1 +b2 MxM MxM +b1 +b2 ReLU ReLU MxMX2 CONV operation

14. Abstract backgrounds

15. DAIR.AI

16. Gradient Backgrounds

17. Community Contributions

18. S=1 S=2 Striding in CONV

19. NxNx192 NxNx64 NxNx32 NxNx128 NxNx192 1x1 Same 3x3 Same 5x5 Same MaxPool Same s=1 Inception Module

20. Retraining w/o expansion t-1 t

21. No- Retraining w/ expansion Partial Retraining w/ expansion

22. No-Retraining w/ expansion Partial Retraining w/ expansion t-1 t t t-1

23. No-Retraining expansion t Partial Retraining expansion t t-1 Retraining expansion t-1 t t-1

24. Size #bed ZIP Wealth Family? Walk? School PRICE ŷ X Y X Ŷ = 0 Ŷ = 1 How does NN work (Insprired from Coursera) Logistic Regression Basic Neuron Model

25. Size $ Size $ Linear regression ReLU(x)

26. I V 128*128*1 128*128*1 CONV 1 CONV 2 CONV 4 CONV 3 CONV 5 CONV 6 CONV 7 I1 128*128*1 ENcoder De coder V1 E ncoder Decoder 128*128*1 Train ing

27. Large NN Med NN Small NN SVM,LR etc η Amount of Data Why does Deep learning work?

28. a [1] 1 a [1] 2 a [1] 3 Input Hidden Output X = A [0] a [1] 4 a [2] A [1] A [2] X Ŷ One hidden layer neural network

29. a [1] 1 a [1] 2 x [1] a [2] x [2] x [2] x [3] x [1] Neural network templates

30. a [1] 1 a [1] 2 x [1] a [2] x [2] x [2] x [3] x [1] Neural network templates

31. Train Valid Test x1 x2 x1 x2 x1 x2 Train-Dev-Test vs. Model fitting Underfitting Good fit Overfitting

32. x [2] x [3] x [1] a [L] x1 x2 r= 1 x1 x2 DropOut Normalization

33. w1 w1 w2 J w1 w2 J w1 w2 w2 Before Normalization After Normalization Early stopping Dev Train Err it.

34. x 1 x 2 w[1] w[2] w[L-2] w[L-1] w[L] FN TN TP FP Deep neural networks Understanding Precision & Recall

35. w1 w2 SGD B GD w1 w2 SGD Batch vs. Mini-batch Gradient Descent B atch Gradient Descent vs. SGD

36. x [2] x [3] x [1] p [1] p [2] Softmax Prediction with 2 outputs

37. Miscellaneous

38. 3 64 16 16 32 32 64 128 128 256 256 128+256 128 1 64+128 64 32+64 32 16+32 16 16 Convolution 3x3 Max Pooling 2x2 Convolution 1x1 Skip connection Up Sampling 2x2 Block copied Dropout 0.1 Dropout 0.2 Dropout 0.3

39. Conv3-32 Conv3-32 Conv3-32 Max-Pool Conv3-32 Conv3-128 Max-Pool Conv3-64 Conv3-64 Max-Pool Input Conv Conv Max-Pool Max-Pool FC Layer1 Softmax FC Layer2 Layer3 Layer4 Layer1 Layer2 Layer3 Layer4 Input Conv3-32 Conv3-32 Conv3-32 Max-Pool Conv3-32 Conv3-128 Conv3-64 Conv3-64 Max-Pool Layer1 Layer2 Layer3 FC-512 Output Max-Pool FC-512 Output

40. Previous layer 1x1 convolutions 1x1 convolutions 3x3 convolutions 1x1 convolutions 5x5 convolutions 3x3 max pooling 1x1 convolutions Filter concatenation

41. Previous layer 1 x3 conv, 1 padding 1x5 conv, 2 padding 1x3 conv, 1 padding 1 x7 conv, 3 padding Filter concatenation 1x3 conv, 1 padding 1x3 conv, 1 padding

42. Input Conv Max-Pool Max-Pool Max-Pool Inception Inception Max-Pool Conv Max-Pool Conv Inception Inception Inception Inception Inception Inception Inception Avg-Pool Conv FC FC Softmax Avg-Pool Conv FC Softmax Avg-Pool Conv FC FC Softmax Auxiliary Classifier Auxiliary Classifier Input ConvTranspose2d Max-Pool Conv Max-Pool

43. Previous layer 1x1 conv. 1x1 conv. 3x3 conv. 1x1 conv. 3x3 conv. Pool 1x1 conv. Filter concatenation 3x3 conv. Previous layer 1x1 conv. 1x1 conv. 1x1 conv. 3x3 conv. Pool 1x1 conv. Filter concatenation 1x3 conv. 3x1 conv. 1x3 conv. 3x1 conv. ( a ) ( b )

44. R1 R2 R3 R1 R2 R3 R1 R1 R1 R2 Stacked layers Previous input x F(x) y=F(x) Stacked layers Previous input x F(x) y=F(x)+x x identity +

45. Input Conv Avg-Pool Dense Block 2 Dense Block 3 Conv Avg-Pool Conv Dense Block 1 Avg-Pool FC Softmax Transition layers

46. 3x3 conv ( a ) add identity 3x3 conv 5x5 conv 3x3 avg identity 3x3 avg 3x3 avg 3x3 conv 5x5 conv add add add add Filter concatenation h i h i-1 ... h i+1 h i h i-1 ... 7x7 conv 5x5 conv 7x7 conv 3x3 max 5x5 conv 3x3 avg add add add identity 3x3 avg 3x3 max 3x3 conv add add Filter concatenation h i+1 ( b )

47. 14*14 28*28 56*56 112*112 224*224 =

48. 1 1 2 4 5 6 7 8 3 2 1 1 2 3 4 6 8 3 4 Max(1,1,5,6) = 6 Image Representation Y X Pooling performed with a 2x2 kernel and a stride of 2