Introduction to Capsule Networks (CapsNets)

CapsNets are a hot new architecture for neural networks, invented by Geoffrey Hinton, one of the godfathers of deep learning.
You can view this presentation on YouTube at: https://youtu.be/pPN8d0E3900

NIPS 2017 Paper:
* Dynamic Routing Between Capsules,
* by Sara Sabour, Nicholas Frosst, Geoffrey E. Hinton
* https://arxiv.org/abs/1710.09829

The 2011 paper:
* Transforming Autoencoders
* by Geoffrey E. Hinton, Alex Krizhevsky and Sida D. Wang
* https://goo.gl/ARSWM6

CapsNet implementations:
* Keras w/ TensorFlow backend: https://github.com/XifengGuo/CapsNet-Keras
* TensorFlow: https://github.com/naturomics/CapsNet-Tensorflow
* PyTorch: https://github.com/gram-ai/capsule-networks

Book:
Hands-On Machine with Scikit-Learn and TensorFlow
O'Reilly, 2017
Amazon: https://goo.gl/IoWYKD

Github: https://github.com/ageron
Twitter: https://twitter.com/aureliengeron

1. Capsule Networks Aurélien Géron, November 2017 https://youtu.be/pPN8d0E3900

2. NIPS 2017 Paper Dynamic Routing Between Capsules by Sara Sabour, Nicholas Frosst, Geoffrey E. Hinton October 2017: https://arxiv.org/abs/1710.09829

3. Computer Graphics Rectangle x=20 y=30 angle=16° Triangle x=24 y=25 angle=-65° Instantiation parameters Image R endering

4. Inverse Graphics Instantiation parameters Image Inverse r endering Rectangle x=20 y=30 angle=16° Triangle x=24 y=25 angle=-65°

5. Capsules Capsule activations Image Inverse rendering = =

6. Activation vector: Capsules Length = estimated probability of presence Orientation = object’s estimated pose parameters = =

7. Squash( u ) = Capsules = = Convolutional Layers Reshape Squash || u || 2 1 + || u || 2 u || u ||

8. Equivariance = =

9. Equivariance = =

10. A hierarchy of parts Boat x=22 y=28 angle=16°

11. A hierarchy of parts Rectangle x=20 y=30 angle=16° Triangle x=24 y=25 angle=-65° Boat x=22 y=28 angle=16°

12. A hierarchy of parts Rectangle x=20 y=30 angle=-5° Triangle x=26 y=31 angle=137° House x=22 y=28 angle=-5°

13. Primary Capsules = = Primary Capsules

14. Predict Next Layer’s Output = = Primary Capsules

15. Predict Next Layer’s Output = = Primary Capsules

16. Predict Next Layer’s Output = = One transformation matrix W i , j per part/whole pair ( i , j ). û j | i = W i , j u i Primary Capsules

17. Predict Next Layer’s Output = = Primary Capsules

18. Predict Next Layer’s Output = = Primary Capsules

19. Compute Next Layer’s Output = = Predicted Outputs Primary Capsules

20. Routing by Agreement = = Predicted Outputs Primary Capsules Strong a greement!

21. The rectangle and triangle capsules should be routed to the boat capsules. Routing by Agreement = = Predicted Outputs Primary Capsules Strong agreement!

22. Clusters of Agreement

23. Clusters of Agreement Mean

24. Clusters of Agreement Mean

25. Clusters of Agreement Mean

26. Clusters of Agreement Mean

27. Clusters of Agreement Mean

28. Routing Weights = = Predicted Outputs Primary Capsules b i , j =0 for all i , j

29. Routing Weights = = Predicted Outputs Primary Capsules 0.5 0.5 0.5 0.5 b i , j =0 for all i , j c i = softmax( b i )

30. Compute Next Layer’s Output = = Predicted Outputs s j = weighted sum Primary Capsules 0.5 0.5 0.5 0.5

31. Compute Next Layer’s Output = = Predicted Outputs Primary Capsules 0.5 0.5 0.5 0.5 s j = weighted sum v j = squash( s j )

32. Actual outputs of the next layer capsules (round #1) Compute Next Layer’s Output = = Predicted Outputs Primary Capsules 0.5 0.5 0.5 0.5 s j = weighted sum v j = squash( s j )

33. Actual outputs of the next layer capsules (round #1) Update Routing Weights = = Predicted Outputs Primary Capsules A greement

34. Actual outputs of the next layer capsules (round #1) Update Routing Weights = = Predicted Outputs Primary Capsules Agreement b i , j += û j | i . v j

35. Actual outputs of the next layer capsules (round #1) Update Routing Weights = = Predicted Outputs Primary Capsules Agreement b i , j += û j | i . v j Large

36. Actual outputs of the next layer capsules (round #1) Update Routing Weights = = Predicted Outputs Primary Capsules Disa greement b i , j += û j | i . v j Small

37. Compute Next Layer’s Output = = Predicted Outputs Primary Capsules 0.2 0.1 0.8 0.9

38. Compute Next Layer’s Output = = Predicted Outputs s j = weighted sum Primary Capsules 0.2 0.1 0.8 0.9

39. Compute Next Layer’s Output = = Predicted Outputs Primary Capsules s j = weighted sum v j = squash( s j ) 0.2 0.1 0.8 0.9

40. Actual outputs of the next layer capsules (round #2) Compute Next Layer’s Output = = Predicted Outputs Primary Capsules 0.2 0.1 0.8 0.9

41. Handling Crowded Scenes = = = =

42. Handling Crowded Scenes = = = = Is this an upside down house?

43. Handling Crowded Scenes = = = = House Thanks to routing by agreement, the ambiguity is quickly resolved (explaining away). Boat

44. Classification CapsNet || ℓ 2 || Estimated Class Probability

45. Training || ℓ 2 || Estimated Class Probability To allow multiple classes, minimize margin loss: L k = T k max(0, m + - || v k || 2 ) + λ (1 - T k ) max(0, || v k || 2 - m - ) T k = 1 iff class k is present In the paper: m - = 0.1 m + = 0.9 λ = 0.5

46. Training Translated to English: “ If an object of class k is present, then || v k || 2 should be no less than 0.9. If not, then || v k || 2 should be no more than 0.1.” || ℓ 2 || Estimated Class Probability To allow multiple classes, minimize margin loss: L k = T k max(0, m + - || v k || 2 ) + λ (1 - T k ) max(0, || v k || 2 - m - ) T k = 1 iff class k is present In the paper: m - = 0.1 m + = 0.9 λ = 0.5

47. Regularization by Reconstruction || ℓ 2 || Feedforward Neural Network Decoder Reconstruction

48. Regularization by Reconstruction || ℓ 2 || Feedforward Neural Network Decoder Reconstruction Loss = margin loss + α reconstruction loss The reconstruction loss is the squared difference between the reconstructed image and the input image. In the paper, α = 0.0005.

49. A CapsNet for MNIST (Figure 1 from the paper)

50. A CapsNet for MNIST – Decoder (Figure 2 from the paper)

51. Interpretable Activation Vectors (Figure 4 from the paper)

52. Pros Reaches high accuracy on MNIST, and promising on CIFAR10 Requires less training data Position and pose information are preserved (equivariance) This is promising for image segmentation and object detection Routing by agreement is great for overlapping objects (explaining away) Capsule activations nicely map the hierarchy of parts Offers robustness to affine transformations Activation vectors are easier to interpret (rotation, thickness, skew…) It’s Hinton! ;-)

53. Not state of the art on CIFAR10 (but it’s a good start) Not tested yet on larger images (e.g., ImageNet): will it work well? Slow training, due to the inner loop (in the routing by agreement algorithm) A CapsNet cannot see two very close identical objects This is called “crowding”, and it has been observed as well in human vision Cons

54. Implementations Keras w/ TensorFlow backend: https://github.com/XifengGuo/CapsNet-Keras TensorFlow: https://github.com/naturomics/CapsNet-Tensorflow PyTorch: https://github.com/gram-ai/capsule-networks

55. Amazon: https://goo.gl/IoWYKD Twitter: @aureliengeron github.com/ageron