predictive maintenace using machine learning.docx

presentation about Machine learning

1. Abstract
The emergence of the Fourth Industrial Revolution coincided with the development
and spread of Internet of Things (IoT) technologies, which enabled their integration
with industrial equipment.
By using IoT sensors integrated with factory equipment, it is possible to collect
massive amounts of data about the condition of these industrial equipment and
machines, such as their operating conditions, temperature, and vibrations. When this
data is properly processed, we have obtained important information about the
production process. It can also continuously monitor the condition of this vital
equipment and systems.
Recently, there has been growing interest in exploiting this data in maintenance
operations.
Using this data, machine learning-based systems are being created to predict the
failure of various components, thus avoiding downtime and equipment damage
through proactive maintenance.
This strategy, known as predictive maintenance (PdM), offers industries advantages
such as extending the life of components, reducing additional costs, saving time, and
maintaining production.
-------------------------------------------------------------------------------------------------
In this study, the 3W database developed by Petrobras was used, which aims to
describe all oil and gas losses within its operating unit.
In the first step of this work,
- The data will be prepared and cleaned for the following operations.
- Then, a genetic algorithm is applied to select features. This step aims to reduce the
number of unnecessary features, reducing the complexity, resources, and time
required in the prediction process.
- Next, the prediction process is performed. At this stage, the random forest algorithm
is also used to predict the equipment condition.
As a result, we demonstrated the power of both the genetic algorithm and random
forest for predicting equipment condition, achieving 95-99% accuracy in the testing
phase.
-------------------------------------------------------------------------------------------------
Contents of Presentation
This Presentation consists of the following:

2. - Section One: Reviews introduction to the strategies of maintenance.
- Section Two: This section will introduce reviews of the datasets that will be used.
- Section Three: This section will include the methodology used for feature selection
and explain the genetic algorithm that will be used to implement feature selection.
- Section Four: This section will discuss the methodology used in predictive
maintenance and equipment condition prediction by discussing and implementing the
Random Forest algorithm.
- Section Five: This section discusses the results of both the feature selection and
predictive maintenance processes.
- Section Five: This section discusses the results of both the feature selection and
predictive maintenance processes.
- Section Six: This section concludes and suggests some points for future research.
-------------------------------------------------------------------------------------------------
Sec 1 Introduction
1.1 Introduction
With the emergence of the Fourth Industrial Revolution (Industry 4.0) and the
development and proliferation of Internet of Things (IoT) technologies, it has become
possible to collect massive amounts of data about factory equipment. For example,
using these devices integrated with factory equipment, data such as equipment
operating conditions, status, temperature, and vibration are obtained. Proper
processing and evaluation of this data helps make important decisions about the
production process and ensures the continued operation of this equipment.
In industrial production environments, equipment maintenance has a critical impact
on cost reduction, planning ability, and sustained production capacity. In particular,
unplanned breakdowns during production periods cause high costs, unplanned
downtime, and potential additional collateral damage, such as delayed customer
contract execution. Unnecessary and inappropriate maintenance activities can lead to
lost time, additional costs, and most importantly, serious consequences for customers.
For this reason, maintenance activities must be implemented according to plan and
when necessary to achieve the desired benefits.
-------------------------------------------------------------------------------------------------
Industries follow different maintenance approaches for their machinery, depending on
their specific needs and constraints. There are:
• Reactive maintenance.
• Preventive maintenance.
• Predictive maintenance (PdM).
-------------------------------------------------------------------------------------------------

3. 1.2 Research Problem
Price fluctuations and challenging environments in the oil sector have driven
companies to improve their productivity. Continuous efforts are being made to
improve key performance indicators by enhancing efficiency, reliability, and reducing
operating costs. This work aims to present predictive maintenance strategies for oil
and gas equipment.
This predictive maintenance framework enables safer operations by providing
continuous diagnostics of processes and equipment and predicting failure times. It
increases plant availability, reduces verification effort by providing real-time,
documented verification without interruption to the process, and enables virtually zero
downtime by providing real-time predictive maintenance information.
-------------------------------------------------------------------------------------------------
1.3 Research Question
Is it possible to develop a predictive model based on variables and data related to
industrial equipment in the oil and gas industry to predict future equipment failures?
What stages, techniques, algorithms, and variables should be considered in the model
to create a consistent framework focused on this modeling process, taking into
account the need to apply it to real datasets for predictive maintenance (PdM) in an
industrial setting?
-------------------------------------------------------------------------------------------------
1.4 Research Objectives
With the advent of the Fourth Industrial Revolution and the increasing use of artificial
intelligence in the manufacturing sector, the main trend in maintenance has become
PdM. It has become increasingly important in industrial sectors with the aim of
reducing maintenance costs and achieving sustainable operational management.
This study aims to transition from traditional, reactive maintenance strategies to an
intelligent, data-driven predictive maintenance (PdM) framework.
1. Build predictive models
To develop and implement a predictive maintenance model using advanced ML
algorithms to accurately predict failures in industrial machinery, thereby preventing
unplanned downtime and equipment breakdowns.
2. Compare ML algorithms
To systematically evaluate and compare the performance of multiple machine
learning algorithms in the context of industrial fault prediction.
3. Design scalable PdM framework

4. To design an adaptable and scalable predictive maintenance framework that can serve
as a foundational blueprint for future PdM projects across different types of industrial
equipment and systems.
4. Optimize for real-world deployment
To optimize the final predictive model by balancing high accuracy with operational
practicality, ensuring it is both effective and efficient enough for real-world industrial
deployment.
-------------------------------------------------------------------------------------------------
1.5 Methodology
- This thesis will begin by examining and extracting information from the datasets
used in the thesis.
- Next, the research will examine genetic algorithms and their use in feature selection.
This process aims to reduce complexity and save processing time.
- Next, we will discuss the algorithm that predicts equipment condition, Random
Forest. The dataset will be divided into two parts: a training part and a testing part.
-------------------------------------------------------------------------------------------------

5. Sec 2 Background
2.1 Offshore Oil Wells
Offshore oil platforms have evolved significantly in recent years, becoming gigantic
structures that can accommodate hundreds of people at a time. Nowadays, there are
different artificial lift techniques, such as beam pumps, gas lift, and electric
submersible pumps, but the scope of this work considers only offshore naturally
flowing wells.
Natural flow wells are those in which the reservoir pressure is sufficient to produce
hydrocarbons at a normal rate without the need for any additional energy. This can
occur in both offshore and onshore wells.
-------------------------------------------------------------------------------------------------
This type of well tends to use less equipment, and therefore less instrumentation,
control loops, and automation. Figure 1 shows a schematic diagram illustrating an
offshore scenario. Oil and gas flow from the reservoir through production pipelines,
then through a production line to a platform.
A Permanent Downhole Gauge (PDG) and a Temperature and Pressure Transducer
(TPT) are devices that contain pressure and temperature sensors, respectively. The
PDG remains fixed in a certain position of the production tubing, and the TPT is part
of the subsea Christmas tree. The well safety valve (DHSV) and production choke
(PCK) are two valves used to control the flow of carbohydrates.
-------------------------------------------------------------------------------------------------
2.2 Data Analysis
Since 2010, Petrobras (One of the largest oil producers in the world) has been
developing a massive database to describe all its losses due to equipment failures in
the oil and gas sector within its operating unit in Rio de Janeiro (OU-Rio).

6. And in mid-2017, Petrobras conceived a project, entitled monitoring of specialized
alarms, to develop a new automated system for detecting and classifying many types
of undesirable events in offshore naturally flowing wells and collecting it in database
called 3W.
-------------------------------------------------------------------------------------------------
The 3W dataset consists of real, simulated, which simulated data was obtained from
the OLGA Dynamic Multiphase Flow Simulator, and hand-drawn data of oil wells
during operation, so it's called 3W.
The data shows instances of the oil well during normal operation and when undesired
events in the oil well occur. This is shown through sensor readings extracted from
eight monitored variables:
1. Pressure at the Permanent Downhole Gauge (PDG) measured with (Pa) unit.
2. Pressure at the Temperature and Pressure Transducer (TPT) measured with (Pa)
unit.
3. Temperature at the TPT measured with (Co) unit.
4. Pressure upstream of the Production Choke Valve (PCK) measured with (Pa) unit.
5. Temperature downstream of the PCK measured with (Co) unit.
6. Pressure downstream of gas lift choke (CKGL) measured with (Pa) unit.
7. Temperature downstream of CKGL measured with (Co) unit.
8. Gas lift flow rate measured with (m3/s) unit.
-------------------------------------------------------------------------------------------------
The data were collected from 21 different wells, with the oldest event occurring in
April 2012 and the most recent in June 2018. The data are divided into folders
according to the event type, and there is a total of 1,984 instances with a total of
15,872 variables.
Every event in the dataset is a continuous sequence of observations with three states:
normal, faulty transient and faulty steady state. These states were created to allow
early detection of a given failure event. The units used in this dataset include Pascal
[Pa], standard cubic meters per second [sm3=s], and degrees in Celsius [C].
-------------------------------------------------------------------------------------------------
The selected types of undesirable events in the 3W dataset include the eight fault
types as follows:
1. Abrupt increase of basic sediment & water
2. Spurious Closure of DHSV
3. Severe Slugging.
4. Flow instability

7. 5. Rapid productivity loss.
6. Quick restriction in PCK.
7. Scaling in PCK.
8. Hydrate in production line.
-------------------------------------------------------------------------------------------------
Event Real Simulate
d
Hand-drawnTotal
0 - Normal operation 597 - - 597
1 - Abrupt increase of BSW5 114 10 129
2 - Spurious closure of DHSV22 16 - 38
3 - Severe slugging 32 74 - 106
4 - Flow instability 344 - - 344
5 - Rapid productivity loss12 439 - 451
6 - Quick restriction in PCK6 215 - 221
7 - Scaling in PCK 4 - 10 14
8 - Hydrate in production line3 81 - 84
Event Real TransientSteady-stateTotal
0 - Normal operation 995679
1
- - 9956791
1 - Abrupt increase of BSW3359177814 5870 118294
2 - Spurious closure of DHSV5265188388 16615 158680
3 - Severe slugging - - 569152 569152
4 - Flow instability - - 2462076 2462076
5 - Rapid productivity loss3223231815810147 361998
6 - Quick restriction in PCK343876252 12951 54212
7 - Scaling in PCK 12526257728821 271708
8 - Hydrate in production line8567 79255 2900 91091
-------------------------------------------------------------------------------------------------
Sec 2 Background
3.1 Introduction
A "feature" or "attribute" refers to an aspect of the data. Usually before collecting
data, features are specified or chosen. So that, Feature Selections is the process of
selecting the best feature among all the features because some of them are not useful
in constructing the clusters: some features may be redundant or irrelevant thus not
contributing to the learning process.
wherein a subset of the features available from the data are selected for application of
a learning algorithm. This subset contains the smallest number of features that
contribute most to increased accuracy, while excluding the remaining unimportant
features that increase complexity and processing time.

8. -------------------------------------------------------------------------------------------------
Although many modified algorithms are available, we proposed population-based
evolutionary algorithms "Genetic Algorithms (GAs)" to provide remedies for these
drawbacks by avoiding local optimum and improving the selection process itself.
- The nature of the optimization model does not need to be known. This makes GAs
very interesting for complex problems or for users inexperienced in gradient-based
optimization techniques.
- The optimization model and its constraints do not have to be continuous or even real
values. No simplification of a problem is necessary to accommodate it to a particular
algorithm.
- It is readily available and easily implemented.
-------------------------------------------------------------------------------------------------
Genetic Algorithms
GA was influenced by Darwin’s evolutionary theory, which comes quite parallel to
the way animals and plants survive in nature and the way they transfer their genes
from one generation to the next. It simulates the process of natural selection which
means those species that can adapt to changes in their environment can survive and
reproduce and go to the next generation.
-------------------------------------------------------------------------------------------------
Solution to a problem solved by genetic algorithm uses an evolutionary process (it
evolved). Algorithm begins with a set of solutions (represented by chromosomes)
called population. Solutions from one population are taken and used to form a new
population. It is an algorithm that depends on the number of populations. Every
parameter denotes a gene, and each solution denotes a chromosome.
Then it uses a fitness (objective) function to assess the fitness of each participant in
the population. For enhancing selected solutions, with a selection procedure, the best
solutions are chosen at random. This operator has a better chance of choosing the
optimal solutions since the likelihood is related to fitness.
The probability of avoiding local optimal solutions is also increased by the possibility
of choosing a weak solution and applying crossover and mutation operators.
Although the genetic algorithm is a random algorithm, it is reliable and capable of
calculating the globally optimal solution to a given problem
-------------------------------------------------------------------------------------------------

9. Figure: Flowchart of genetic algorithm
The two distinct elements in the GA are individuals and populations.
Individuals: An individual is a single solution. Everyone in the population is called a
string or chromosome, in analogy to chromosomes in natural systems. The
chromosome, which is the raw ‘genetic’ information that the GA deals.
Populations: The population is the set of individuals currently involved in search. The
two important aspects of population used in Genetic Algorithms are:
1. The initial population generation.
2. The population size.
-------------------------------------------------------------------------------------------------
genetic algorithms rely on a population of candidate solutions. So that, the population
size, which is usually a user-specified parameter, is one of the important factors that
mainly affect the scalability and performance of genetic algorithms.
Ex, small population sizes might lead to premature convergence and yield substandard
solutions. On the other hand, large population sizes lead to unnecessary expenditure
of valuable computational time.
-------------------------------------------------------------------------------------------------
Steps of Genetic Algorithm
1- Initialization
The first step in the genetic algorithm is to create an initial population that will evolve
over time. These group together potential solutions to a given problem. This allows

10. for greater diversity. At this stage, it’s not a question of finding the right solution. It’s
all about identifying enough solutions capable of responding to the problem. In fact,
the more varied the initial population, the more likely it is that the best possible
solutions can be devised.
Then we will Encode of chromosomes, The process can be performed using bits,
numbers, trees, arrays, lists or any other objects.
In binary encoding, every chromosome is a string of bits (0 or 1), Once the problem is
encoded in a chromosomal manner, then we choose fitness measure for discriminating
good solutions from bad ones has been chosen.
-------------------------------------------------------------------------------------------------
2- Fitness Function
Once the population has been created, it’s time to fitness measures for discriminating
against good solutions from bad, we evaluate each individual according to his or her
ability to solve the problem. This phase of the genetic algorithm is complex, as it is
sometimes difficult to compare two individuals with each other in particularly true for
multi-criteria problems, where the optimal solution depends on several parameters,
without one being better than the other.
In every iteration, the individuals are evaluated based on their fitness scores which are
computed by the fitness function. Individuals who achieve a better fitness score
represent better solutions and are more likely to be chosen to crossover and passed on
to the next generation.
In summary, the fitness function in genetic algorithms is a measure of the quality or
suitability of a potential solution.
-------------------------------------------------------------------------------------------------
In this work, because the variables aren't related to each other and it is difficult to find
an easy way to calculate the fitness function, this stage was implemented by
Distributed Evolutionary Algorithms in Python (DEAP) framework. Framework
DEAP is a powerful evolutionary computation framework in Python designed for
rapid prototyping and testing of genetic algorithms (GA), genetic programming (GP),
and other evolutionary optimization techniques, where it provides tools to implement
evolutionary algorithms efficiently. It can easily calculate the fitness function and
combine variables to calculate the best possible solutions.
-------------------------------------------------------------------------------------------------
3- Selection
Selection of individuals for the next generation, either to reproduce or to live on,
relies heavily on the evaluation function. How heavily is dependent on which
selection technique you use. It allocates more copies of those solutions with higher
fitness values and thus imposes the survival-of-the-fittest mechanism on the candidate
solutions. The main idea of selection is to prefer better solutions to worse ones, and
many selection Ways to accomplish this idea, including:

11. - roulette-wheel selection.
- stochastic universal selection.
-------------------------------------------------------------------------------------------------
a- Roulette-wheel selection
In roulette wheel selection, the individuals are mapped to contiguous segments of a
circular, such that each individual’s segment is equally sized to its fitness. A random
number is generated and the individual whose segment spans the random number is
selected. The process repeats until the desired number of individuals is obtained
(called mating population). This technique is analogous to a roulette wheel with each
slice proportionally sized to the fitness, where a common selection approach assigns a
probability of selecting Pi to each individual i based on its fitness value. The
probability Pi for each individual is defined by
P [Individual i is chosen] =
Fi
∑
j=1
PopSize
Fj
where Fi equals the fitness of individual i.
-------------------------------------------------------------------------------------------------
b- Stochastic Universal Sampling
Stochastic Universal Sampling (SUS) developed by Baker is a single-phase sampling
algorithm with minimum spread and zero bias. Instead of a single selection pointer
employed in roulette wheel methods, SUS uses N equally spaced pointers, where N is
the number of selections required. The population is shuffled randomly and a single
random number pointer1 in the range [0, 1/N] is generated.
The N individuals are then chosen by generating the N pointers, starting with pointer1
and spaced by 1/N, and selecting the individuals whose fitness spans the positions of
the pointers.
If et(i) is the expected number of trials of individual i, |_et(i) _| is the floor of et(i) and
|¯et(i) ¯| is the ceiling, an individual is thus guaranteed to be selected a minimum of
times |_et(i) _| and no more than |¯et(i) ¯|, thus achieving minimum spread. In
addition, as individuals are selected entirely on their positions in the population, SUS
has zero bias. For these reasons, SUS has become one of the most widely used
selection algorithms in current GA.
-------------------------------------------------------------------------------------------------

12. Figure demonstrates the stochastic universal sampling. The individuals are mapped to
contiguous segments of a line, such that each individual’s segment is equal in size to
its fitness exactly as in roulette wheel selection. Equally spaced pointers are placed
over the line as many as there are individuals to be selected (N). For X individuals (N
= X) to be selected, the distance between the pointers is 1/X. Figure shows the
selection for the sample of the random number in the range [0, 1/X].
-------------------------------------------------------------------------------------------------
4- Crossover
A crossover operator in computer science refers to a general operator used in genetic
algorithms to create a new solution by selecting parameters or genes from two parent
solutions. The goal is to inherit beneficial traits from both parents.
The crossover operators are classified into categories such as standard crossovers,
binary crossovers
-------------------------------------------------------------------------------------------------
a- Standard Crossovers
1-Point Crossover
It is one of the simple crossover techniques used for random GA applications. This
crossover uses the single point fragmentation of the parents and then combines the
parents at the crossover point to create the offspring or child.
1-Point crossover first selects two parents used for crossover and then randomly
selects any crossover point pi (i = 0 to n-1).
This results in two offspring, each carrying some genetic information from both
parents. In the above figure, the point between 4th and 5th gene is selected as
crossover point at which genes are exchanged between the parents.
-------------------------------------------------------------------------------------------------
2- K-POINT CROSSOVER
It uses the random crossover point to combine the parents same as per 1-Point
crossover. To provide a great combination of parents it selects more than one
crossover point to create the offspring or child.

13. K-Point Crossover first selects the two parents used for crossover and then randomly
select K crossover points P1i to Pk-1i (i = 0 to n-1). Two offspring are created by
combining the parents at crossover point.
-------------------------------------------------------------------------------------------------
b- BINARY CROSSOVERS
1- Random Respectful Crossover (RRC)
RRC is based on the idea of respecting the common alleles (gene values) shared by
both parents while randomly selecting differing alleles from either parent.
In this type, we use a list named similarity vector to produce the offspring Sab =
(S1ab…, Sn ab). To calculate similarity vectors, we compare the genes of the two
parents in each position. If the value of the genes in that position is the same, the
value of the similarity vector for that position becomes the value of the genes in that
position. Otherwise, the value of the similarity vector for that position becomes -1.
To calculate the values of the genes for the offspring we do the following.
If the value of the similarity vector in position i is 0, the value of the genes in that
position for both offspring becomes 0.
If the value of the similarity vector in position i is 1, the value of the genes in that
position for both offspring becomes 1.
If the value of the similarity vector in position i is -1, we produce a uniform random
real number w between 0 and 1 to determine what the value of the offspring in that
position would be. If w < 0.5, the value of the gene in that position becomes 1;
otherwise, the value of the gene in that position becomes 0.

14. -------------------------------------------------------------------------------------------------
2- Masked Crossover
We use a mask vector to determine which bits of which parent is inherited by the
offspring. The first step is the duplication of the bits of the parents.
The bits of the first parent are copied to the first offspring and, accordingly, of the
second parent to the second offspring.
-------------------------------------------------------------------------------------------------
5- Mutation
Mutation may be defined as a small random tweak in the chromosome, to get a new
solution, in simple terms, it is process of randomly altering some symbols in a
solution's string, and it is used to maintain and introduce diversity in the genetic
population. It can be performed with different probabilities, such as fixed, adaptive,
etc., and is usually applied with a low probability (pm). In the other hand, a higher
value of mutation probability encourages the exploration of the search space.
However, it can also increase the chances of the genetic algorithm getting stuck in a
suboptimal solution. So that in artificial intelligence, mutations are often used to
generate new solutions in the hope of finding a better solution.
It is often used in conjunction with other operators, such as crossover (recombination)
and selection.
-------------------------------------------------------------------------------------------------
There are many ways to mutate:
a- Bit Flip Mutation

15. In bit-flip mutation, one or several positions are randomly selected, and their values
are flipped. In other words, in the selected positions, if the value of the gene is 1, it
will become 0; and if the value of the gene is 0, it will become 1.
-------------------------------------------------------------------------------------------------
b- Random Resetting Mutation
In random resetting mutation, one or several positions are randomly selected, and a
value is determined for each of those positions from the allowed or valid domain. A
domain could be the positive integers.
-------------------------------------------------------------------------------------------------

16. c- Swap Mutation
In swap mutation, two genes are randomly selected, and the value of those two genes
is swapped.
Figure 3.9 Swap Mutation
-------------------------------------------------------------------------------------------------
4.1 Introduction
Due to the nature of big data, such as complexity, high dimensionality, frequent
variance, and irregularity, it is difficult to automatically uncover useful knowledge
and information from real, unstructured, and complex massive datasets without
preprocessing. Therefore, there is an urgent need to apply ML techniques to big data
to facilitate the handling and exploitation of this data.
ML is a branch of AI focused on building systems by computers to create Models that
learn from data. So, in general, machine learning is about learning to do better in the
future based on what was experienced in the past.
The emphasis of machine learning is on automatic methods. In other words, the goal
is to create learning algorithms that learn automatically from available data without
human intervention or assistance and evolve to achieve desired results.
-------------------------------------------------------------------------------------------------
The basic machine learning process can be divided into three parts.
1. Data Input: Past data or information is utilized as a basis for future decision-making
2. Abstraction: The input data is represented in a broader way through the underlying
algorithm
3. Generalization: The abstract representation is generalized to form a framework for
making decisions.
-------------------------------------------------------------------------------------------------

17. By the learning style, machine learning algorithms can be mainly divided into the
following three types. This taxonomy of machine learning algorithms considers the
training data during the model preparation process.
1.Supervised learning – Also called predictive learning. A machine predicts the
class of unknown objects based on prior class-related information of similar
objects.
2.Unsupervised learning – Also called descriptive learning. A machine finds
patterns in unknown objects by grouping similar objects together.
3.Reinforcement learning – A machine learns to act on its own to achieve the
given goals.
-------------------------------------------------------------------------------------------------
a- Supervised Learning
The major motivation of supervised learning is to learn from past information, this is
the information about the task which the machine must execute, where it is provided
with two sets of data, a training set and a test set. The principle is for the training set
test set learner to “learn” from a set of labeled examples in the training set so that it
can identify unlabeled examples in the test set with the highest possible accuracy. The
final goal is to develop a program, or a procedure that classifies new records (in the
test set) by analyzing the training dataset it has been given to have a class label.
-------------------------------------------------------------------------------------------------
Labelled training data containing past information comes as an input. Based on the
training data, the machine builds a predictive model that can be used on test data to
assign a label for each record in the test data. Supervised machine learning is as good
as the data used to train it. If the training data is of poor quality, the prediction will
also be far from being precise.
-------------------------------------------------------------------------------------------------
b- Unsupervised Learning

18. Unlike supervised learning, in unsupervised learning there is no labelled training data
to learn from. It is use of AI algorithms to analyze data sets that do not contain data
points that have been categorized or labeled in any way, there are we only have
features without corresponding outputs or labels for our dataset. Large amounts of
data are required for unsupervised machine learning, where the objective is to take a
dataset as input and try to find natural groupings or patterns within the data elements
or records.
The purpose of learning the algorithm is to find patterns in the dataset and rate the
data points according to those patterns.
Two subcategories of unsupervised learning issues are clustering and association.
-------------------------------------------------------------------------------------------------
c- Reinforcement learning (RL)
Machines often learn to do tasks autonomously, when a sub-task is accomplished
successfully, a reward is given. When a subtask is not performed correctly, the action
will be no reward. This continues until the machine can complete execution of the
whole task. This process of learning is known as reinforcement learning (RL), where
it is a kind of a policy that depends on the whole history of states, actions, and
rewards and selects the next action to take.