OLIVAW: reaching superhuman strength at Othello

https://www.meetup.com/it-IT/Machine-Learning-Data-Science-Meetup/events/258491250/

1. OLIVAW: reaching superhuman strength at Othello From my Master thesis Deep learning for Othello Computer Science, Sapienza University of Rome Thesis Advisor: Prof. Alessandro Panconesi 19-02-19 Antonio Norelli using the AlphaGo Zero paradigm with Zero budget

2. Why games? Shannon 1949 Turing 1950 McCarthy 1956

3. A satisfactory solution of [chess] will act as a wedge in attacking other problems of a similar nature and of greater significance . Programming a Computer for Playing Chess Philosophical Magazine , Vol 41, No. 314 Shannon 1949

4. Games as a micro world Clear objectives Small set of rules Still interesting complexity

5. Choosing a move Naïf approach

6. Games as trees

7. Games as trees

8. Games as trees

9. Looking into the future Tic-tac- toe Connect 4 Checkers Othello Chess Go

10. Looking into the future Tic-tac- toe Connect 4 Checkers Othello Chess Go

11. Exloring the full tree is impossible  

12. Exloring the full tree is impossible  

13. What the future holds for us?

14. What the future holds for us?

15. Oracle to evaluate intermediate positions

16. Oracle to evaluate intermediate positions -0,42

17. Oracle using game knowledge

18. DeepBlue 1997

19. AlphaGo 2016

20. AlphaGo Zero 2017

21. AlphaGo Zero 2017

22. The AlphaGo Zero paradigm: is it universal?

23. The AlphaGo Zero paradigm: is it universal? Does it scale DOWN in terms of resources?

24. The AlphaGo Zero paradigm: is it universal? Does it scale DOWN in terms of resources? This thesis: Can we reach superhuman strength at Othello with the same paradigm and "normal" computing power?

25. AlphaGo Zero My solution 5000 TPUs ≈1 40 (3) Days of training 30 $ 25 Million Estimated Hardware cost $ 500 Budget

26. AlphaGo Zero My solution 5000 TPUs ≈1 40 (3) Days of training 30 $ 25 Million Estimated Hardware cost $ 500 Budget

27. AlphaGo Zero My solution 5000 TPUs ≈1 40 (3) Days of training 30 $ 25 Million Estimated Hardware cost $ 500

28. AlphaGo Zero My solution 5000 TPUs ≈1 40 (3) Days of training 30 $ 25 Million Estimated Hardware cost $ 500 Budget

29. AlphaGo Zero My solution 5000 TPUs ≈1 40 (3) Days of training 30 $ 25 Million Estimated Hardware cost $ 500 Budget

30. Why Othello ? Simple but still interesting

31. Why Othello ? Simple but still interesting

32. Why Othello ? Simple but still interesting

33. Why Othello ? Simple but still interesting

34. Why Othello ? Simple but still interesting

35. Why Othello ? Well known Simpler but still interesting

36. Why Othello ? Well known Simpler but still interesting Easy to implement

37. OLIVAW algorithm

38. OLIVAW algorithm

39. OLIVAW algorithm

40. OLIVAW algorithm

41. OLIVAW training process

42. OLIVAW training process +0,24 [[0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0.03 0. 0. 0. 0. 0. ] [0.15 0. 0. 0. 0. 0. 0. 0. ] [ 0.64 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0.18 0. 0. 0. 0. 0. ]]

43. OLIVAW training process +0,24 [[0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0.03 0. 0. 0. 0. 0. ] [0.15 0. 0. 0. 0. 0. 0. 0. ] [ 0.64 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0.18 0. 0. 0. 0. 0. ]]

44. OLIVAW training process +0,24 [[0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0.03 0. 0. 0. 0. 0. ] [0.15 0. 0. 0. 0. 0. 0. 0. ] [ 0.64 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0.18 0. 0. 0. 0. 0. ]]

45. OLIVAW training process -0,03 [[0. 0. 0. 11 0.11 0. 0. 0. 0.12 ] [0. 0.13 0.9 0. 0. 0. 0. 0. ] [0.10 0.11 0. 0. 0. 0. 0. 0. ] [0.12 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0. 0. 0. 0. 0. 0. ] [0. 0. 0.11 0. 0. 0. 0.11 0. ]]

46. OLIVAW training process VS

47. OLIVAW training process VS

48. OLIVAW training process 1. Self-play games generation

49. OLIVAW training process 1. Self-play games generation

50. OLIVAW training process 1. Self-play games generation

51. OLIVAW training process 1. Self-play games generation 2. Neural Net training

52. OLIVAW training process 1. Self-play games generation 2. Neural Net training VS

53. OLIVAW training process 1. Self-play games generation 2. Neural Net training VS

54. OLIVAW training process 1. Self-play games generation 2. Neural Net training 3. Evaluation

55. OLIVAW training process 1. Self-play games generation 2. Neural Net training 3. Evaluation

56. Pseudocode

57. Reaching superhuman strength

58. OLIVAW vs Alessandro Di Mattei 2016-2017 Italian champion

59. OLIVAW vs Alessandro Di Mattei 2016-2017 Italian champion 2-3 Draw – Draw – Defeat – Victory - Defeat 27-11-2018

60. OLIVAW vs Alessandro Di Mattei

61. OLIVAW vs Alessandro Di Mattei 4-0 3-12-2018

62. OLIVAW strength Alessandro Di Mattei

63. OLIVAW vs Di Mattei 03-12 Game 4

64. OLIVAW vs Michele Borassi 2008 World Othello champion

65. OLIVAW Is still training VIDEO TRAILER

66. OLIVAW vs Michele Borassi Best of 3 19 January 2019 16,30 – Dipartimento di Matematica Guido Castelnuovo Sapienza, piazzale Aldo Moro, 5

67. 1-0 OLIVAW wins with black OLIVAW vs Michele Borassi Game 1

68. 1-1 Michele Borassi wins with black OLIVAW vs Michele Borassi Game 2

69. 1-2 Michele Borassi wins with black OLIVAW vs Michele Borassi Game 3

70. Thanks! Any questions? You can find me at [email protected] OLIVAW: reaching superhuman strength at Othello using the AlphaGo Zero paradigm with Zero budget Antonio Norelli