Biological and Artificial Neural Networks

Biological and Artificial Neural Networks

1. Artificial Neural Networks Damian Gordon

2. Contents What we already know… How does the brain work? What are Neurons? How do Artificial Neural Networks work? How are Artificial Neural Networks trained? Training an ANN to recognize an XOR Gate

3. What we already know…

4. Artificial Intelligence Biologically-inspired Systems Neural Networks

5. Artificial Intelligence Artificial Neural Networks

6. Artificial Intelligence

7. How does the brain work?

8. How does the brain work? The three main parts of the brain are: Cerebrum – Responsible for thinking, memory, emotions, voluntary movements, and sensory perception. Cerebellum – Controls coordination, balance, and fine motor skills. Brainstem – Regulates automatic functions like breathing, heartbeat, and digestion.

9. How does the brain work? Some people like to think of the cerebrum as being made up of 2 hemispheres, and create cool images like this:

10. How does the brain work? But we shouldn’t get too excited about that sort of thing, the brain exhibits a phenomenon called plasticity, which means the brain can change and adapt in structure and function in response to learning and experience. So, a better way of looking at the Cerebrum is to think about it as being divided into four lobes:

11. How does the brain work? Frontal Lobe : Controls thinking, decision-making, problem-solving, and voluntary movements. Parietal Lobe : Processes touch, temperature, pain, and spatial awareness. Temporal Lobe : Important for hearing, memory, and language comprehension. The Wernicke’s Occipital Lobe : Primarily responsible for vision, it contains the visual cortex, which interprets visual signals from the eyes.

12. How does the brain work? Neurons are the building blocks of the brain, acting like tiny messengers that send and receive information.

13. The brain has around 86 billion neurons, which communicate using electrical and chemical signals to control everything we think, feel, and do. How does the brain work?

14. What are Neurons?

15. What are Neurons? Neurons (or nerve cells ) are cells that fire electric signals called action potentials across a neural network in the nervous system. They are located in the brain and spinal cord and help to receive and conduct impulses.

16. What are Neurons? Sensory Neurons (Afferent Neurons) carry sensory information from sensory receptors (e.g., skin, eyes, ears) to the central nervous system. They detect stimuli like light, sound, temperature, and pressure. Motor Neurons (Efferent Neurons) transmit signals from the CNS to muscles and glands, causing movement or secretion. They control voluntary and involuntary muscles. Interneurons (Association Neurons) are found within the CNS, these neurons connect sensory and motor neurons. They process information and play a crucial role in reflexes, decision-making, and higher cognitive functions.

17. What are Neurons? Neurons can send two types of signals: Excitatory : An excitatory signal makes a neuron more likely to fire an action potential (electrical impulse). Inhibitory : An inhibitory signal makes a neuron less likely to fire an action potential.

18. What are Neurons? Neurons work as follows: A neuron receives input from multiple other neurons. If the combined input (sum of excitatory and inhibitory signals) reaches a certain threshold, the neuron fires an action potential. If the input is below the threshold, the neuron does not fire.

19. What are Neurons? Let’s think about how the work using an exaggerated example:

20. What are Neurons? We are in the forest:

21. What are Neurons? We are in the forest: Hear a sound in the bushes

22. What are Neurons? We are in the forest: Hear a sound in the bushes Saw a tiger here last week

23. What are Neurons? We are in the forest: Hear a sound in the bushes Saw a tiger here last week Tiger footprints in the sand

24. What are Neurons? We are in the forest: Hear a sound in the bushes Saw a tiger here last week Tiger footprints in the sand + + +

25. What are Neurons? We are in the forest: Hear a sound in the bushes Saw a tiger here last week Tiger footprints in the sand + + +

26. What are Neurons? Let’s look at a different example:

27. What are Neurons? We are in the forest:

28. What are Neurons? We are in the forest: Hear a sound in the bushes

29. What are Neurons? We are in the forest: Hear a sound in the bushes Never seen a tiger here

30. What are Neurons? We are in the forest: Hear a sound in the bushes Never seen a tiger here No footprints in the sand

31. What are Neurons? We are in the forest: Hear a sound in the bushes Never seen a tiger here No footprints in the sand + - -

32. What are Neurons? We are in the forest: Hear a sound in the bushes Never seen a tiger here No footprints in the sand + - -

33. What are Neurons? So a brain is really something like this:

34. What are Neurons? So a brain is really something like this:

35. How do Artificial Neural Networks work?

36. How do Artificial Neural Networks work? Artificial Neural Networks (ANNs) are inspired by biological neurons but are implemented using mathematical models in software or hardware.

37. How do Artificial Neural Networks work? They process data by passing it through layers of interconnected artificial neurons (also called nodes or units), mimicking how the human brain learns patterns and relationships.

38. How do Artificial Neural Networks work? An ANN has three layers: Input Layer Hidden Layer(s) Output Layer

39. How do Artificial Neural Networks work? Input Layer The first layer receives raw data (e.g., images, text, numbers). Each neuron in this layer represents one feature of the input. For example, if we had an image made up of 25 pixels, and we had 25 nodes in the input layer, each neuron might represent a pixel value.

40. How do Artificial Neural Networks work? Hidden Layer(s) These layers process the input node using weighted connections. More hidden layers mean the network can learn more complex patterns (deep learning networks have multiple layers).

41. How do Artificial Neural Networks work? Hidden Layer(s) Each neuron in a hidden layer performs: Weighted sum of inputs : We need to figure out how much importance we give to each input neuron (so we might give some neurons more weight than others), and we can add in a bias value, just in case the sum of weights total to zero. Activation function: We also need to figure out what level of input from each neuron is enough to send an activation ( action potential ) onto the next layer. It could be as simple as the sum of the values, or something more complicated.

42. How do Artificial Neural Networks work? Hidden Layer(s) Weighted sum of inputs: Hear a sound in the bushes Never seen a tiger here No footprints in the sand + - - Should we really just add these up, or does hearing something get more weight than no footprints? + BIAS

43. How do Artificial Neural Networks work? Hidden Layer(s) Activation function: Hear a sound in the bushes Never seen a tiger here No footprints in the sand + - - The activation function decides whether or not to send this signal onto the next layer, we can decide based on a simple sum or more complex calcuations f (x) + BIAS

44. How do Artificial Neural Networks work? Hidden Layer(s) Activation function: STEP FUNCTION This is a simple STEP FUNCTION, if the sum of weights in great than a specific value the signal is sent.

45. How do Artificial Neural Networks work? Hidden Layer(s) Activation function: STEP FUNCTION SEND ACTIVATION DON’T SEND ACTIVATION

46. How do Artificial Neural Networks work? Hidden Layer(s) Activation function: STEP FUNCTION Sum is below required value Sum is above required value SEND ACTIVATION DON’T SEND ACTIVATION

47. How do Artificial Neural Networks work? Hidden Layer(s) Activation function: BUT THERE ARE OTHERS WE COULD APPLY…

48. How do Artificial Neural Networks work? Hidden Layer(s) Activation function: SIGMOID FUNCTION This is a SIGMOID FUNCTION, it’s like a dimmer switch, if it’s dark, the light goes FULLY ON, if it’s dim the light goes PARTIALLY ON, and if it’s bright, the light is OFF.

49. How do Artificial Neural Networks work? Hidden Layer(s) Activation function: LINEAR FUNCTION This is a LINEAR FUNCTION, it takes the value the is input into it, and passes that value onto the next neuron in the chain without modifying it.

50. How do Artificial Neural Networks work? Hidden Layer(s) Activation function: TANH FUNCTION This is a HYPERBOLIC TANGENT FUNCTION, if the input value is summed to be very positive SEND STRONG ACTIVATION, if it is very negative DO NOT SEND ACTIVATION, if it is in between SEND WEAK ACTIVATION.

51. How do Artificial Neural Networks work? Output Layer The final layer produces the final result (classification, prediction, etc.). For example, in a binary classification task (e.g., spam vs. not spam), the output might be 0 or 1. For example, in multi-class classification, outputs are probabilities for different categories.

52. How do Artificial Neural Networks work? https://www.mladdict.com/neural-network-simulator

53. How are Artificial Neural Networks trained?

54. How are Artificial Neural Networks trained? The first steps before training begins are: The Artificial Neural Network is set up. The nodes are assigned random weights to start off with, sometimes they all give 0.5.

55. How are Artificial Neural Networks trained? We also set a learning rate , which controls how fast the ANN updates its knowledge while learning: If the learning rate is too high, the AI learns too fast, making wild guesses and missing the right answer. If the learning rate is too low, the AI learns too slowly, taking forever to improve.

56. How are Artificial Neural Networks trained? They are trained in three steps: A. Forward Propagation B. Loss Function C. Backpropagation & Weight Updates

57. How are Artificial Neural Networks trained? Forward Propagation Input data is fed into the network. It passes through the layers, undergoing weighted sums and activation functions. The output is generated and compared to the actual target value

58. How are Artificial Neural Networks trained? B. Loss Function Measures how far the predicted output is from the actual result. Common loss functions: Mean Squared Error (MSE) for regression tasks. Cross-Entropy Loss for classification.

59. How are Artificial Neural Networks trained? C. Backpropagation & Weight Updates Backpropagation adjusts the weights to minimize the error. The gradient of the loss function is calculated using the chain rule (calculus). The weights are updated based on Loss and the learning rate. This process repeats for many iterations (epochs) until the network achieves good accuracy.

60. Training an ANN to recognize an XOR Gate

61. Training an ANN to recognize an XOR Gate

62. Training an ANN to recognize an XOR Gate In 1 In 2 H 1 H 2 H 3 O 1 https://www.mladdict.com/neural-network-simulator

63. Training an ANN to recognize an XOR Gate #include < stdio.h > #include < stdlib.h > #include < math.h > // Learning rate #define LEARNING_RATE 0.5

64. Training an ANN to recognize an XOR Gate The program starts by including necessary header files: < stdio.h > for input and output operations like printf . < stdlib.h > for standard utility functions. < math.h > for mathematical operations like exponentiation (exp), which is used in the activation function. A constant learning rate of 0.5 is defined. This controls how much the weights are adjusted during each training step. A high learning rate can lead to instability, while a low value might slow down training.

65. Training an ANN to recognize an XOR Gate // Sigmoid activation function double sigmoid(double x) { return 1 / (1 + exp(-x)); }

66. Training an ANN to recognize an XOR Gate The sigmoid function is implemented to transform input values into a range between 0 and 1. This function is useful in neural networks because it introduces non-linearity, allowing the model to learn complex patterns.

67. Training an ANN to recognize an XOR Gate // Derivative of sigmoid double sigmoid_derivative (double x) { return x * (1 - x); }

68. Training an ANN to recognize an XOR Gate The derivative of the sigmoid function is used during the backpropagation process to update weights efficiently. The formula ensures the error is properly propagated backward through the network.

69. Training an ANN to recognize an XOR Gate int main() { // Training data (XOR problem) double input[4][2] = { {0, 0}, {0, 1}, {1, 0}, {1, 1} };

70. Training an ANN to recognize an XOR Gate The XOR dataset is defined with four input pairs: (0,0) → expected output: 0 (0,1) → expected output: 1 (1,0) → expected output: 1 (1,1) → expected output: 0 This is a well-known problem that cannot be solved using a simple perceptron, requiring a neural network with a hidden layer.

71. Training an ANN to recognize an XOR Gate // Initialize weights and biases with random values double hidden_weights [2][3] = {{0.5, -0.2, 0.3}, {-0.3, 0.8, -0.5}}; double hidden_bias [3] = {0.1, -0.1, 0.2}; double output_weights [3] = {0.6, -0.4, 0.7}; double output_bias = 0.2;

72. Training an ANN to recognize an XOR Gate Initialize Weights and Biases Random initial weights and biases are assigned: Hidden layer weights (2 input neurons × 3 hidden neurons). Hidden layer biases (one for each hidden neuron). Output layer weights (connecting 3 hidden neurons to 1 output neuron). Output bias (affecting the final output). These values determine how the network computes its output before training begins.

73. Training an ANN to recognize an XOR Gate int epochs = 10000; // Number of training iterations

74. Training an ANN to recognize an XOR Gate The network is trained for 10,000 epochs (iterations over the entire dataset). Each epoch involves feeding all training samples into the network, calculating errors, and updating weights.

75. Training an ANN to recognize an XOR Gate for (int i = 0; i < 4; i ++) { // Forward pass - Hidden layer double hidden_layer [3]; for (int j = 0; j < 3; j++ ) { hidden_layer [j] = sigmoid(input[ i ][0] * hidden_weights [0][j] +input[ i ][1] * hidden_weights [1][j] + hidden_bias [j]); }

76. Training an ANN to recognize an XOR Gate The training process iterates over all four input-output pairs. Each example is passed through the network, and errors are computed. For each training example, the hidden layer neurons compute their activations: Each neuron calculates a weighted sum of the inputs. A bias term is added. The sigmoid function is applied to introduce non-linearity. This process transforms the input data into a more useful representation for the next layer.

77. Training an ANN to recognize an XOR Gate // Forward pass - Output layer double output = sigmoid( hidden_layer [0] * output_weights [0] + hidden_layer [1] * output_weights [1] + hidden_layer [2] * output_weights [2] + output_bias );

78. Training an ANN to recognize an XOR Gate Once the hidden layer activations are computed, the output neuron performs a similar process: It takes the weighted sum of hidden layer outputs. A bias is added. The sigmoid function is applied again to produce a final predicted output.

79. Training an ANN to recognize an XOR Gate // Compute error double error = expected_output [ i ] - output; total_error += error * error; // Backpropagation - Output layer double output_delta = error * sigmoid_derivative (output);

80. Training an ANN to recognize an XOR Gate The error is calculated as the difference between the expected and predicted output. The squared error is accumulated to monitor the overall performance. To adjust weights, backpropagation starts by computing the gradient (delta) for the output neuron. This gradient determines how much the neuron contributed to the total error.

81. Training an ANN to recognize an XOR Gate // Backpropagation - Hidden layer double hidden_delta [3]; for (int j = 0; j < 3; j++ ) { hidden_delta [j] = output_delta * output_weights [j] * sigmoid_derivative ( hidden_layer [j]); }

82. Training an ANN to recognize an XOR Gate The error is then propagated back to the hidden layer. Each hidden neuron’s contribution to the error is calculated using the output delta and its associated weight.

83. Training an ANN to recognize an XOR Gate // Update weights and biases for (int j = 0; j < 3; j++ ) { output_weights [j] += LEARNING_RATE * output_delta * hidden_layer [j]; hidden_weights [0][j] += LEARNING_RATE * hidden_delta [j] * input[ i ][0]; hidden_weights [1][j] += LEARNING_RATE * hidden_delta [j] * input[ i ][1]; hidden_bias [j] += LEARNING_RATE * hidden_delta [j]; }

84. Training an ANN to recognize an XOR Gate Using gradient descent, weights and biases are updated to reduce the error in future iterations: O utput weights are updated using the computed output delta. Hidden layer weights are updated using the computed hidden delta. Biases are also adjusted accordingly. This step ensures that the network gradually improves its accuracy.

85. Training an ANN to recognize an XOR Gate output_bias += LEARNING_RATE * output_delta ; } // Print error every 1000 epochs if (epoch % 1000 == 0) { printf ("Epoch %d, Error: %.6f\n", epoch, total_error / 4); } }

86. Training an ANN to recognize an XOR Gate To monitor training progress, the average error is printed every 1000 epochs. This helps in verifying whether the network is learning effectively.

87. Training an ANN to recognize an XOR Gate // Testing the trained network printf ("\ nTesting the trained neural network:\n"); for (int i = 0; i < 4; i ++) { double hidden_layer [3];

88. Training an ANN to recognize an XOR Gate // Forward pass - Hidden layer for (int j = 0; j < 3; j++ ) { hidden_layer [j] = sigmoid(input[ i ][0] * hidden_weights [0][j] + input[ i ][1] * hidden_weights [1][j] + hidden_bias [j]); }

89. Training an ANN to recognize an XOR Gate // Forward pass - Output layer double output = sigmoid( hidden_layer [0] * output_weights [0] + hidden_layer [1] * output_weights [1] + hidden_layer [2] * output_weights [2] + output_bias );

90. Training an ANN to recognize an XOR Gate After training, the network is tested on the XOR dataset: Each input pair is passed through the network. The computed output is displayed to check if the network has correctly learned the XOR function. Since XOR is non-linearly separable, the network needs the hidden layer to correctly classify the inputs.

91. Training an ANN to recognize an XOR Gate printf ("Input: (%.0f, %.0f) -> Output: %.4f\n", input[ i ][0], input[ i ][1], output); } return 0; }

92. Training an ANN to recognize an XOR Gate Finally, the program exits successfully.