Associative memory network

This presentation discusses about various techniques of Associative memory network under Neural Networks

1. Dr . C.V. Suresh Babu Professor Department of IT Hindustan Institute of Science & Technology Associative memory network

2. Action Plan Associative Memory Networks - Introduction to auto associative memory network - Auto associative memory architecture - Auto associative memory training & testing algorithm - Introduction to hetero associative memory network - Hetero associative memory architecture - Hetero associative memory training & testing algorithm Quiz at the end of session

3. Associative Memory Networks These kinds of neural networks work on the basis of pattern association, which means they can store different patterns and at the time of giving an output they can produce one of the stored patterns by matching them with the given input pattern. These types of memories are also called Content-Addressable Memory CAM .

4. Auto Associative Memory - Architecture This is a single layer neural network in which the input training vector and the output target vectors are the same. As shown in the following figure, the architecture of Auto Associative memory network has ‘n’ number of input training vectors and similar ‘n’ number of output target vectors.

5. Auto Associative Memory – Training Algorithm For training, this network is using the Hebb or Delta learning rule. Step 1 − Initialize all the weights to zero as w ij = 0, i = 1 to n, j = 1 to n Step 2 − Perform steps 3-4 for each input vector. Step 3 − Activate each input unit as follows − x i = s i (i = 1 to n ) Step 4 − Activate each output unit as follows − y j = s j (j = 1 to n ) Step 5 − Adjust the weights as follows − w ij (new) = w ij (old) + x i y j

6. Auto Associative Memory – Testing Algorithm Step 1 − Set the weights obtained during training for Hebb’s rule. Step 2 − Perform steps 3-5 for each input vector. Step 3 − Set the activation of the input units equal to that of the input vector. Step 4 − Calculate the net input to each output unit j = 1 to n n y inj = ∑ x i w ij i=1 Step 5  − Apply the following activation function to calculate the output y j = f(y inj ) = +1 if y inj > 0 - 1 if y inj ⩽ 0

7. Hetero Associative Memory Similar to Auto Associative Memory network, this is also a single layer neural network. The weights are determined so that the network stores a set of patterns.

8. Hetero Associative Memory - Architecture As shown in the following figure, the architecture of Hetero Associative Memory network has ‘n’ number of input training vectors and ‘m’ number of output target vectors .

9. Hetero Associative Memory – Training Algorithm For training, this network is using the Hebb or Delta learning rule. Step 1 − Initialize all the weights to zero as w ij = 0, i = 1 to n, j = 1 to m Step 2 − Perform steps 3-4 for each input vector. Step 3 − Activate each input unit as follows − x i = s i (i = 1 to n ) Step 4 − Activate each output unit as follows − y j = s j (j = 1 to m) Step 5 − Adjust the weights as follows − w ij (new) = w ij (old) + x i y j

10. Hetero Associative Memory – Testing Algorithm Step 1 − Set the weights obtained during training for Hebb’s rule. Step 2 − Perform steps 3-5 for each input vector. Step 3 − Set the activation of the input units equal to that of the input vector. Step 4 − Calculate the net input to each output unit j = 1 to m n y inj = ∑ x i w ij i=1 Step 5  − Apply the following activation function to calculate the output +1 if y inj > 0 y j = f(y inj ) = 0 if y inj = 0 - 1 if y inj < 0

11. Quiz - Questions 1. What is the other name of associative memory? 2. In which associative memory network, the input training vector and the output target vectors are the same? a) auto b) hetero c) iterative d) noniterative 3. In which associative memory network, the input training vector and the output target vectors are not the same? a) auto b) hetero c) iterative d) noniterative 4. For which algorithm does the associative memory networks use the Hebb or Delta learning rule? a) training b) testing c) processing d) none 5. For which algorithm does the associative memory networks set the activation of the input units equal to that of the input vector . a ) training b) testing c) processing d) none

12. Quiz - Answers 1. What is the other name of associative memory? Content-Addressable Memory ( CAM) 2. In which associative memory network, the input training vector and the output target vectors are the same? a) auto 3. In which associative memory network, the input training vector and the output target vectors are not the same? b) hetero 4. For which algorithm does the associative memory networks use the Hebb or Delta learning rule? a) training 5. For which algorithm does the associative memory networks set the activation of the input units equal to that of the input vector . b ) testing

13. Action Plan Associative Memory Networks (Cont…) - Introduction to iterative auto associative network - Introduction to bidirectional associative network - BAM operation - BAM stability and storage capacity Quiz at the end of session Assignment – 2: Write a detailed note on iterative auto associative memory.

14. Iterative Auto Associative Network Net does not respond to the input signal with the stored target pattern. Respond like stored pattern. Use the first response as input to the net again. Iterative auto associative network recover original stored vector when presented with test vector close to it. It is also known as recurrent auto associative networks.

15. Bidirectional Associative Memory (BAM) Bidirectional associative memory (BAM), first proposed by Bart Kosko, is a hetero associative network. It associates patterns from one set, set A, to patterns from another set, set B, and vice versa. Human memory is essentially associative. We attempt to establish a chain of associations, and thereby to restore a lost memory.

16. BAM Operation

17. BAM Operation (Cont…) The correlation matrix is the matrix product of the input vector X, and the transpose of the output vector Y T . The BAM weight matrix is the sum of all correlation matrices, that is , where M is the number of pattern pairs to be stored in the BAM.

18. BAM Operation (Cont…) The input vector X (p) is applied to the transpose of weight matrix W T to produce an output vector Y(p). Then , the output vector Y(p) is applied to the weight matrix W to produce a new input vector X(p+1). This process is repeated until input and output vector become unchanged, or in other words, the BAM reaches stable state.

19. Stability and Storage Capacity of the BAM The BAM is unconditionally stable. The maximum number of associations to be stored in the BAM should not exceed the number of neurons in the smaller layer . The more serious problem with the BAM is incorrect convergence. In fact, a stable association may be only slightly related to the initial input vector.

20. Quiz - Questions 1. What is the other name of iterative auto associative networks? 2 . BAM is a ------------ associative network. 3. What has to be created for each pattern pair in order to develop BAM? 4. The major issue with BAM is ------------ . 5. Who first proposed BAM?

21. Quiz - Answers 1. What is the other name of iterative auto associative networks? Recurrent auto associative networks 2 . BAM is a ------------ associative network. Hetero 3. What has to be created for each pattern pair in order to develop BAM? Correlation matrix 4. The major issue with BAM is ------------ . Incorrect convergence 5. Who first proposed BAM? Bart Kosko

22. Action Plan Associative Memory Networks (Cont…) - Introduction to Hopfield networks - Introduction to Discrete Hopfield networks - Discrete Hopfield networks training & testing algorithm - Energy function evaluation - Introduction to Continuous Hopfield networks Quiz at the end of session

23. Hopfield Networks The Hopfield network represents an auto-associative type of memory . Hopfield neural network was invented by Dr. John J. Hopfield in 1982. It consists of a single layer which contains one or more fully connected recurrent neurons.

24. Discrete Hopfield Network The network has symmetrical weights with no self-connections i.e., w ij = w ji and w ii = 0 . Architecture Following are some important points to keep in mind about discrete Hopfield network − - This model consists of neurons with one inverting and one non- inverting output. - The output of each neuron should be the input of other neurons but not the input of self.

25. Discrete Hopfield Network (Cont…) - Weight/connection strength is represented by w ij . - Weights should be symmetrical, i.e. w ij = w ji The output from Y 1 going to Y 2 , Y i and Y n have the weights w 12 , w 1i and w 1n respectively. Similarly, other arcs have the weights on them.

26. Discrete Hopfield Network – Training Algorithm During training of discrete Hopfield network, weights will be updated. As we know that we can have the binary input vectors as well as bipolar input vectors. Hence , in both the cases, weight updates can be done with the following relation: Case 1 − Binary input patterns For a set of binary patterns s p , p = 1 to P Here, s p = s 1 p , s 2 p ,..., s i p ,..., s n p Weight Matrix is given by P w ij = ∑ [2s i (p )−1][2s j (p)−1 ] for i ≠ j p=1

27. Discrete Hopfield Network – Training Algorithm Case 2 − Bipolar input patterns For a set of binary patterns s p , p = 1 to P Here, s p = s 1 p , s 2 p ,..., s i p ,..., s n p Weight Matrix is given by P w ij = ∑ [s i (p )][s j (p )] for i ≠ j p=1

28. Discrete Hopfield Network – Testing Algorithm Step 1 − Initialize the weights, which are obtained from training algorithm by using Hebbian principle. Step 2 − Perform steps 3-9, if the activations of the network is not consolidated. Step 3 − For each input vector X , perform steps 4-8. Step 4 − Make initial activation of the network equal to the external input vector X as follows − y i = x i for i = 1 to n Step 5 − For each unit Y i , perform steps 6-9.

29. Discrete Hopfield Network – Testing Algorithm Step 6 − Calculate the net input of the network as follows − y ini =x i + ∑ y j w ji j Step 7 − Apply the activation as follows over the net input to calculate the output − 1 if y ini > θ i y i = y i if y ini = θ i 0 if y ini < θ i Here θ i is the threshold. Step 8 − Broadcast this output y i to all other units. Step 9 − Test the network for conjunction.

30. Energy Function Evaluation An energy function is defined as a function that is bonded and non-increasing function of the state of the system . Energy function E f ⁡, ⁡also called Lyapunov function determines the stability of discrete Hopfield network, and is characterized as follows − n n n n E f = − 1 / 2 ∑ ∑ y i y j w ij − ∑ x i y i + ∑ θ i y i i=1 j=1 i=1 i=1

31. Continuous Hopfield Network Model − The model or architecture can be build up by adding electrical components such as amplifiers which can map the input voltage to the output voltage over a sigmoid activation function . Energy Function Evaluation n n n n n y i E f = 1 / 2 ∑ ∑ y i y j w ij − ∑ x i y i + 1 / λ ∑ ∑ w ij g ri ∫ a − 1 (y)dy i=1 j=1 i=1 i=1 j=1 0 j≠i j≠ i Here λ is gain parameter and g ri input conductance.

32. Quiz - Questions 1. The Hopfield network is an ---------- associative type of memory . 2. Hopfield consists of a -------- layer which contains one or more fully connected recurrent neurons. a) single b) double c) triple d) linear 3. Which principle is used to initialize weights in testing algorithm? 4. What is the other name of energy function? 5. Continuous Hopfield network has --------- as a continuous variable. a) weight b) time c) bias d) none

33. Quiz - Answers 1. The Hopfield network is an ---------- associative type of memory . Auto 2. Hopfield consists of a -------- layer which contains one or more fully connected recurrent neurons. a) single 3. Which principle is used to initialize weights in testing algorithm? Hebbian principle 4. What is the other name of energy function? Lyapunov function 5. Continuous Hopfield network has --------- as a continuous variable. b) time