Neuroevolution in neural networks with dynamic neurons

Congress oral presentation

1. Structural and Parametric
Evolution of Continuous-time
Recurrent Neural Networks
Cesar Gomes Miguel, EP/USP
Carolina Feher da Silva, ICB/USP
Marcio Lobo Netto, EP/USP
10th Brazilian Symposium on Neural Networks
Salvador, Bahia
2008
University of São Paulo
Polytechnic School
Electronic Systems Department

2. 2
Introduction
Learning in Neural Networks:

3. 3
Introduction
Learning in Neural Networks:
Parametric Synaptic weights, bias, etc...

4. 4
Introduction
Learning in Neural Networks:
Parametric Synaptic weights, bias, etc...
Structural Add or remove new nodes or links
(modifies the topology)

5. 5
Introduction
Learning in Neural Networks:
Parametric Synaptic weights, bias, etc...
Structural Add or remove new nodes or links
(modifies the topology)
In supervised learning problems it is possible to rely
on traditional optimization methods for parametric
learning, e.g.:
● Error-back propagation (gradient descent)
● Truncated-Newton methods

6. 6
Introduction
● Combinatorial optimization could also be used for
structural supervised learning (but probably never
attempted before)

7. 7
Introduction
● Combinatorial optimization could also be used for
structural supervised learning (but probably never
attempted before)
● For general RL domains we need different
approaches. Evolutionary methods, such as Genetic
Algorithms, are often a good choice.

8. 8
Neuroevolution
What is neuroevolution?
Genetic
Algorithms
Artificial Neural
Networks
+

9. 9
Neuroevolution
What is neuroevolution?
Genetic
Algorithms
Artificial Neural
Networks
+
● several encodings
● crossover operator
● speciation

10. 10
Neuroevolution
What is neuroevolution?
Genetic
Algorithms
Artificial Neural
Networks
+
● several encodings
● crossover operator
● speciation
● binary
● rate-coding
● spiking

11. 11
Neuroevolution
What is neuroevolution?
Genetic
Algorithms
Artificial Neural
Networks
+
● several encodings
● crossover operator
● speciation
● binary
● rate-coding
● spiking
Neuroevolution = 90% GA + 10% ANN

12. 12
Neuroevolution
NEAT (NeuroEvolution of Augmenting Topologies)
●Developed by Stanley (2004) at UT Austin;
●Most widely applied and analysed method;
●Can be extended to evolve any graph-like structure;

13. 13
Neuroevolution
Main features:
●Efficient genetic encoding, allowing reliable genetic
recombination between different topologies;

14. 14
Neuroevolution
Main features:
●Efficient genetic encoding, allowing reliable genetic
recombination between different topologies;
●Chromosomes are clustered by genetic similarity
(maintains diversity through speciation);

15. 15
Neuroevolution
Main features:
●Efficient genetic encoding, allowing reliable genetic
recombination between different topologies;
●Chromosomes are clustered by genetic similarity
(maintains diversity through speciation);
●Mutations add new nodes/links and modify
parametric attributes (weights, bias, ...);

16. 16
Neuroevolution
Main features:
●Efficient genetic encoding, allowing reliable genetic
recombination between different topologies;
●Chromosomes are clustered by genetic similarity
(maintains diversity through speciation);
●Mutations add new nodes/links and modify
parametric attributes (weights, bias, ...);
●Evolution starts from the smallest possible topology
and complexifies as needed;

17. 17
Neuroevolution
Current limitations in the field:
●Most neuroevolutionary methods evolve traditional
(discrete) neural models (NEAT is not an exception);

18. 18
Neuroevolution
Current limitations in the field:
●Most neuroevolutionary methods evolve traditional
(discrete) neural models (NEAT is not an exception);
●In continuous RL domains the agent is continuously
interacting with the environment, so it would be
more appropriate to incorporate a neuron model that
is time-dependent;

19. 19
Continuous-time Neuron
●A well known model is the continuous-time recurrent
neural network (a.k.a CTRNN):

20. 20
Continuous-time Neuron
●A well known model is the continuous-time recurrent
neural network (a.k.a CTRNN):
●Usually implemented as the control system in
Evolutionary Robotics, with the help of GAs for
parametric learning (Beer 1992).

21. 21
Evolving CTRNNs
Using NEAT to evolve CTRNNs would improve the
control system over traditional ANNs in
continuous RL domains.
Hypothesis

22. 22
Evolving CTRNNs
Using NEAT to evolve CTRNNs would improve the
control system over traditional ANNs in
continuous RL domains.
Double pole balancing task (a long used
benchmark test among different neuroevolution
methods)
How to verify?
Hypothesis

23. 23
Experiment
Double Pole Non-Markovian (DPNV):

24. 24
DPNV
Many ramifications:

25. 25
DPNV
Many ramifications:
Partially observable:
the poles velocities
are not shown to the
CTRNN

26. 26
DPNV
How to assign fitness?
Following Gruau (1994) methodology:
Such that:
Penalizes high oscillations

27. 27
DPNV
●The best performing network is evaluated in a
separate generalization test (with 625 different
initial conditions);

28. 28
DPNV
●The best performing network is evaluated in a
separate generalization test (with 625 different
initial conditions);
●It must balance for at least 200 different conditions
to be accepted as a valid solution;

29. 29
DPNV
●The best performing network is evaluated in a
separate generalization test (with 625 different
initial conditions);
●It must balance for at least 200 different conditions
to be accepted as a valid solution;
●The set of differential equations (CTRNNs and the
cart-pole's system) was integrated using RK4 (h =
0.01).

30. 30
Results
Comparing with previous methods and NEAT itself:

31. 31
Results
Comparing with previous methods and NEAT itself:

32. 32
Results
Comparing with previous methods and NEAT itself:

33. 33
Results
Comparing with previous methods and NEAT itself:

34. 34
Conclusion
● NEAT-CTRNN performed significantly better
than traditional NEAT (p < 0.001%);

35. 35
Conclusion
● NEAT-CTRNN performed significantly better
than traditional NEAT (p < 0.001%);
● Elitism can be harmful in the DPNV task;

36. 36
Conclusion
● NEAT-CTRNN performed significantly better
than traditional NEAT (p < 0.001%);
● Elitism can be harmful in the DPNV task;
● New strategies to benchmark new methods (the
DPNV is not a challenge for today's standards);

37. 37
Conclusion
● NEAT-CTRNN performed significantly better
than traditional NEAT (p < 0.001%);
● Elitism can be harmful in the DPNV task;
● New strategies to benchmark new methods (the
DPNV is not a challenge for today's standards);
● More theoretical results regarding the search
space;

38. 38
Thank you!