Neocognitron Model in deep Learning in Introduction Part

Neocognitron Model in deep Learning in Introduction Part which is used for hand written number recognization process

1. Module 1 Introducing Deep Learning The Neocognitron

2. The Neocognitron Prof. Kunihiko Fukushima, Japan. He has special interests lie in modeling neural networks of the higher brain functions, especially, the mechanism of the visual system. He invented the " Neocognitron " for deformation invariant pattern recognition, and the "Selective Attention Model", which can recognize and segment overlapping objects in the visual fields. One of his recent research interests is in modeling neural network for active vision in the brain. He is the author of many books on neural networks, including "Neural Networks and Information Processing", "Neural Networks and Self-Organization", and "Physiology and Bionics of the Visual of the Visual System".

3. What is Neocognitron ? The Neocognitron is a hierarchical multilayered neural network proposed by Professor Kunihiko Fukusima for handwritten character recognition. At present there are many different versions of the neocognitron . Two original basic versions proposed by Professor Fukushima differ in used learning principle mainly: Learning without a teacher Learning with a teacher

4. The main advantage of neocognitron is its ability to recognize correctly not only learned patterns But also patterns which are produced from them by using of partial shift, rotation or another type of distortion.

5. Example: On this simple example we will demonstrate abilities of the neocognitron at recognition of presented patterns. The black-box in this example contains neocognitron network which can distinguish between two different types of patterns (between digit zero and digit one). For the learning of the network, we have used patterns shown below. Patterns 0 and 1 used for learning

6. By the learning of the neocognitron to distinguish between these two types of patterns we have created two different categories in the network. In the future the network will respond on every presented pattern with a pair of values. Each of these values is a measures of belonging of presented pattern into one of two created categories. Click on one of the prepared patterns. The network processes it and assigns it to one of the categories. Notice, that the network assigns patterns which have not been presented during learning to the correct category, too. These patterns were produced by distortion of patterns used for learning.

7. Structure of the neocognitron arises from a hierarchy of extracted features. One appropriate stage of the neocognitron is created for each stage of the hierarchy of extracted features. The network however contains one additional stage, labeled as stage 0, which is not used, in contrast to higher stages, for feature extraction.

8. Total number of stages of the neocognitron depends on the complexity of recognized patterns. The more complex recognized patterns are, the more stages of hierarchy of extracted features we need and the higher number of stages of the neocognitron is. Network structure - Stages

9. Network structure - Layers

10. Network structure layer it is obvious that four types of layers exist in the neocognitron . Stage 0 always consists of only one input layer. All higher consist of one S-layer, one V-layer and one C-layer. In Network structure, we have also established ordinarily used notation of layers in the neocognitron . We will use this notation, described in below table, in the following text as well. Symbol Denotes UO Input layer US1 S- layer - in the 1-th stage of the network UVI V-layer - in the 1-th stage of the network UCI C-layer - in the 1-th stage of the network

11. Each layer in the neocognitron consists of certain number of cell planes of the same type. Input layer is exception from this rule. For the input layer the term cell plane is not established. Number of cell planes in each S – layer and C – layer depends on the number of features extracted in corresponding stage of the network. Each V-layer always consists of only one cell plane. Structure of the network from Network structure layer after drawing of cell planes from which the particular layers are assembled is below diagram

12. Network structure – Cell planes

13. Now we have come to the ground of the neocognitron which is cell. The neocognitron is made of large amount of cells of several distinct types which are organized in cell planes, layers and stages. All the cells, regardless of their type, process and generate analog values. From the below figure it is obvious that each S-Plane, V-Plane, C-Plane and input layer consists of array of cells of the certain type. Size of cell arrays is the same for all cell planes in one layer and it decreases with increasing of the network stage. Each C-Plane in the highest stage of the network contains only one cell. Its output value indicates a measure of belonging of presented pattern into the category represented by this cell. Size of cell array in each V-Plane is the same as size of cell arrays in S-Planes in the same stage of the network.

14. Network structure – Cells

15. It is obvious that four types of cells exist in the neocognitron – receptor cells, S- Cells V-Cells C-Cells On the following pages we will explain S-Cell, V-Cell and C-Cell function in detail

16. S – Cell funciton

17. Connection areas of the V – Cell

18. Connection areas of the C – Cell

19. Tolerance of feature shifts

20. Weights in the neocognitron

21. Weights sharing is the next term being connected with weights. By the term we designate the fact that all cells in one cell plane use the same weights for connections leading from cells in their connection areas. By the means of weight sharing it is guaranteed that all cells from one cell plane always extract the same feature. Weight Sharing

22. Weights We can split the weights as Weights in the neocognitron figure according to the way which they are adjusted Weights modified by learning a-weights b-weights Fixed weights c-weights d-weights

23. a-Weights a-weights are the first type of weights modified by learning. These weights are used for connections between S-cells and C-cells which belong to their connection areas. Features extracted by S-cells are encoded in these a- weigts . Adjusting of a-weights is performed during learning of the network according to the presented training patterns.

24. b-Weights b-weights are the second type of weights modified by learning. These weights are used for connections between S-cells and corresponding V-cells. Adjusting of b-weights is performed during learning of the network according to the presented training patterns as well.

25. c-Weights Fixed c-weights are used for connection between V-cells and C-cells which belong to their connection areas. Values of C-weights are determined at construction of the network. Those weights are most often set up in such a way that they mostly reduce transfer of information from the periphery of connection area and towards the center of area the degree of reduction decreases.

26. d-Weights Fixed d-weights are used for connection between C-cells and S-cells which belong to their connection areas. As well as c-weights also d-weights are determined at construction of the network and again in such a way so as to reduce transfer of information from periphery of connection areas mostly.

27. THANK YOU