Self Play Networks and Generative Artificial Networks

Self Play Networks and Generative Artificial Networks

1. Unit 2 Self-play networks, generative adversarial networks, learning from concept-drifting data streams. 1

2. Self-play networks What is self play? Self-play is a framework where an agent learns to play a game by playing against itself. That is, when an agent is training, it uses its current and past selves as sparring partners. DeepMind’s  AlphaGo Zero  used this technique. Self-play is used by the  AlphaZero  program to improve its performance in the games of  chess ,  shogi  and  go AlphaGo Zero, the reinforcement learning agent, learned to master Go by playing against itself — AlphaGo Zero. This contrasts with the original AlphaGo which learned against expert human players. 2

3. 3

4. 4

5. Self-play networks Self-play has a long history in the training of game-play agents. Agents have been constructed to play checkers, backgammon, chess, go. In 1959, Arthur Samuel [  Samuel  ] wrote a program that learned to play checkers. The system learned from samples like “board configuration, game outcome”. Samples were collected from human play and self play. In 1992, Gerald Tesauro [  Tesauro  ] created TD-Gammon, a neural network trained to play backgammon at a world-champion level. This NN learned to evaluate a backgammon board configuration  entirely  by self-play. 5

6. Self-play networks Deepmind’s AlphaZero [  Silver, et. al.  ] took this line of research to an extreme by learning to play chess — which is fantastically more complex than checkers — and Go — which is fantastically more complex than chess. If it only played against itself, where did new information come from? How did it know if it was doing well? If I play a game of chess against myself, should I say I did well if I beat myself? But when I beat myself, I also lose to myself. And how could I ever know if I’d do well against someone else? Self-play isn’t completely understood, as far as I can tell. There are no guarantees about performance, for example, and no accepted recipe for it. But people have made it work many times. 6

7. Self-play networks Self-play is revolutionary. First, it eliminates human bias. Consider the original AlphaGo which trained against expert human Go players. Well, by playing against expert human players, AlphaGo’s data was limited and biased. In other words, humans play Go in a human-like way, and that might not be the optimal way to play the game. An agent that plays against itself unleashes creativity that is unconstrained by human bias or assumptions. 7

8. Self-play networks The great benefit of self-play is that you dont need to give direct supervision; all of the information that the agent uses to learn comes from another agent (a clone of itself) acting to maximize its own objectives. The reason AlphaZero could skyrocket so quickly into strategic competence is because, given the fixed rules of the game, the aggressively paralleled algorithm could simulate game episodes much faster than a true human learning the game. 8

9. GENERATIVE ADVERSARIAL NETWORKS A Generative Adversarial Network (GAN) is a deep learning architecture that consists of two neural networks competing against each other in a zero-sum game framework.  The goal of GANs is to generate new, synthetic data that resembles some known data distribution. Generative Adversarial Networks (GANs) are a powerful class of neural networks that are used for  unsupervised learning . It was developed and introduced by Ian J. Goodfellow in 2014. It consists of 2 models that automatically discover and learn the patterns in input data.  GANs are basically made up of a system of two competing neural network models which compete with each other and are able to analyze, capture and copy the variations within a dataset. 9

10. GENERATIVE ADVERSARIAL NETWORKS Components: The two models are known as Generator and Discriminator. Generator network: creates synthetic data Discriminator network: evaluates the synthetic data and tries to determine if it’s real or fake They compete with each other to scrutinize, capture, and replicate the variations within a dataset. GANs can be used to generate new examples that plausibly could have been drawn from the original dataset. 10

11. GENERATIVE ADVERSARIAL NETWORKS Shown below is an example of a GAN. There is a database that has real 100 rupee notes. The generator neural network generates fake 100 rupee notes. The discriminator network will help identify the real and fake notes. 11

12. GENERATIVE ADVERSARIAL NETWORKS What is a Generator? A Generator in GANs is a neural network that creates fake data to be trained on the discriminator. It learns to generate plausible data. The generated examples/instances become negative training examples for the discriminator. It takes a fixed-length random vector carrying noise as input and generates a sample.  12

13. GENERATIVE ADVERSARIAL NETWORKS 13

14. GENERATIVE ADVERSARIAL NETWORKS What is a Discriminator? The Discriminator is a neural network that identifies real data from the fake data created by the Generator. The discriminator's training data comes from different two sources: The real data instances, such as real pictures of birds, humans, currency notes, etc., are used by the Discriminator as positive samples during training. The fake data instances created by the Generator are used as negative examples during the training process. 14

15. GENERATIVE ADVERSARIAL NETWORKS 15

16. GENERATIVE ADVERSARIAL NETWORKS While training the discriminator, it connects to two loss functions. During discriminator training, the discriminator ignores the generator loss and just uses the discriminator loss. In the process of training the discriminator, the discriminator classifies both real data and fake data from the generator. The discriminator loss penalizes the discriminator for misclassifying a real data instance as fake or a fake data instance as real. The discriminator updates its weights through backpropagation from the discriminator loss through the discriminator network. 16

17. GENERATIVE ADVERSARIAL NETWORKS 17

18. GENERATIVE ADVERSARIAL NETWORKS How Do GANs Work? GANs consists of two neural networks. There is a Generator G(x) and a Discriminator D(x). Both of them play an adversarial game. The generator's aim is to fool the discriminator by producing data that are similar to those in the training set. The discriminator will try not to be fooled by identifying fake data from real data. Both of them work simultaneously to learn and train complex data like audio, video, or image files. The Generator network takes a sample and generates a fake sample of data. The Generator is trained to increase the Discriminator network's probability of making mistakes 18

19. GENERATIVE ADVERSARIAL NETWORKS 19

20. GENERATIVE ADVERSARIAL NETWORKS Below is an example of a GAN trying to identify if the 100 rupee notes are real or fake. So, first, a noise vector or the input vector is fed to the Generator network. The generator creates fake 100 rupee notes. The real images of 100 rupee notes stored in a database are passed to the discriminator along with the fake notes. The Discriminator then identifies the notes as classifying them as real or fake. We train the model, calculate the loss function at the end of the discriminator network, and backpropagate the loss into both discriminator and generator models. 20

21. GENERATIVE ADVERSARIAL NETWORKS 21

22. GENERATIVE ADVERSARIAL NETWORKS Mathematical Equation The mathematical equation for training a GAN can be represented as: Here,  G = Generator D = Discriminator Pdata (x) = distribution of real data p(z) = distribution of generator x = sample from Pdata (x) z = sample from P(z) D(x) = Discriminator network G(z) = Generator network 22

23. GENERATIVE ADVERSARIAL NETWORKS 23 STEPS FOR TRAINING GAN Define the problem Choose the architecture of GAN  Train discriminator on real data  Generate fake inputs for the generator  Train discriminator on fake data Train generator with the output of the discriminator

24. GENERATIVE ADVERSARIAL NETWORKS Different types of GANs. Vanilla GANs: Vanilla GANs have a min-max optimization formulation where the Discriminator is a binary classifier and uses sigmoid cross-entropy loss during optimization. The Generator and the Discriminator in Vanilla GANs are multi-layer perceptrons . The algorithm tries to optimize the mathematical equation using stochastic gradient descent. Deep Convolutional GANs (DCGANs): DCGANs support convolution neural networks instead of vanilla neural networks at both Discriminator and Generator. They are more stable and generate better quality images. The Generator is a set of convolution layers with fractional- strided convolutions or transpose convolutions, so it up-samples the input image at every convolutional layer. The discriminator is a set of convolution layers with strided convolutions, so it down-samples the input image at every convolution layer. 24

25. GENERATIVE ADVERSARIAL NETWORKS Conditional GANs: Vanilla GANs can be extended into Conditional models by using extra-label information to generate better results. In CGAN, an additional parameter ‘y’ is added to the Generator for generating the corresponding data. Labels are fed as input to the Discriminator to help distinguish the real data from the fake generated data. Super Resolution GANs: SRGANs use deep neural networks along with an adversarial network to produce higher resolution images. SRGANs generate a photorealistic high-resolution image when given a low-resolution image. 25

26. GENERATIVE ADVERSARIAL NETWORKS APPLICATIONS OF GAN With the help of DCGANs, you can train images of cartoon characters for generating faces of anime characters as well as Pokemon characters. 26

27. GENERATIVE ADVERSARIAL NETWORKS GANs can be trained on the images of humans to generate realistic faces. The faces that you see below have been generated using GANs and do not exist in reality. GANs can build realistic images from textual descriptions of objects like birds, humans, and other animals. We input a sentence and generate multiple images fitting the description.  27

28. GENERATIVE ADVERSARIAL NETWORKS GANs can build realistic images from textual descriptions of objects like birds, humans, and other animals. We input a sentence and generate multiple images fitting the description. 28

29. Data drift in AI Data drift is unexpected and undocumented changes to data structure, semantics, and infrastructure that is a result of modern data architectures. Data stream Streaming data is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of Kilobytes). Learning from Concept-Drifting Data Streams

30. Concept drift in data streams During classification a change in the concept or distribution of dataset over the time is termed as concept drift. Concept drift AI models Concept drift is a type of model drift where the properties of the dependent variable changes. Drift analysis for machine learning In machine learning, model drift means that the machine learning model becomes less and less accurate due to the changes in the statistical properties of the input features, target variable, or relationships among variables. Learning from Concept-Drifting Data Streams.

31. Types of data stream e-Commerce purchase data Social media sentiment analysis Sensor readings from machines Stock exchange data to predict the stock price Credit card transactions for fraud detection

32. 1 B Covariate/feature drift Covariate drift or feature drift happens when p(X) changes but p( y|x ) remains the same. The marginal distribution of the input house features changes, but the conditional distribution of house prices given house features stays the same. TITLE 02 Make a big impact with professional slides, charts, infographics and more. Types of model drift Model drift contains two key categories: concept drift and data drift. Data drift further includes covariates/feature drift and label drift.

33. 2 Label drift Label drift happens when p(y) changes but p( x|y ) remains the same. In the house price prediction example, the house price distribution p(y) could change after when the model was trained previously Types of model drift

34. 3 B Concept drift Concept drift happens when p( y|X ) changes but p(X) remains the same. In the house price prediction example, the conditional probability of housing price given house features p( y|X ) could change Types of model drift

35. One of the main causes of data drift is  sampling mismatch . We use some sampling strategies to find the source data to train our model. The sampling strategy could be  biased and not representative  of the entire population. CAUSES OF MODEL DRIFT

36. CAUSES OF MODEL DRIFT Anomalies  can appear both in the training data and also in the target data. Seasonal effects  can also lead to a model drift. The relationship between multiple variables, for example, house size and house price, might change depending on the month or the season. Data quality  issues can happen more than you would expect. There are a variety of data quality issues such as incorrect input data, incorrect data processing steps, and duplications.

37. DETECTING DRIFT Monitor model performance Monitor distribution changes Monitor descriptive statistics The most straightforward approach to detecting drift is through monitoring model performance metrics. The most common model performance metrics include confusion matrix, accuracy, recall, F1 score, and ROC-AUC We can describe our datasets statistically with measurements like min, max, median, mean, uniqueness, correlation, and others We can include all these descriptive statistics in our model monitoring dashboard to visualize how these statistics change over time. How to monitor machine learning in production? The most straightforward approach to monitoring ML models in production is to  evaluate its performance as real-world data grows, evolves, and diversifies .  There are several statistical tests or hypothesis tests people use to detect the distribution changes statistically such as Population Stability Index, Exact Test, Maximum Mean Discrepancy

38. Tools There are a lot of ML monitoring tools and model drift monitoring tools out there. Here are a few popular tools Evidently   For example,  Amazon SageMaker Model Monitor  “detects data skew by comparing real-world data to a baseline dataset such as a training dataset or an evaluation dataset.”   whylogs Alibi Detect