3Naive Bayes Classifier_Machine Learning.pdf

Sigmoid function,Classification Algorithm in Machine Learning: Decision Trees ,
Ensemble Techniques: Bagging and boosting, Adaboost and gradient boost, Random
Forest,Naïve Bayes Classifier, Support Vector Machines. Performance Evaluation:
Confusion Matrix, Accuracy, Precision, Recall, AUC-ROC Curves, F-Measure

1. Course - Machine Learning
Course code-IT 312
Unit-III-Classification
Topic- Bayes Classifier
Sanjivani Rural Education Society’s
Sanjivani College of Engineering, Kopargaon-423603
(An Autonomous Institute Affiliated to Savitribai Phule Pune University, Pune)
NAAC ‘A’ Grade Accredited, ISO 9001:2015 Certified
Department of Information Technology
(NBA Accredited)
Dr.R.D.Chintamani
Asst. Prof.
1

2. ML- Unit-III Classification Department of IT

Unit-III- Classification
•Syllabus
•Sigmoid function, Classification Algorithm in Machine Learning: Decision
•Trees , Bagging and boosting, Adaboost and gradient boost,Random Forest,
•Naïve Bayes Classifier, Support Vector Machines. Performance Evaluation:
•Confusion Matrix, Accuracy, Precision, Recall, AUC-ROC Curves,
•F-Measure
2

3. Naive Bayes Classifier
Applications of Naive Bayes Classifier -
A)Real time predictions - Naive bayes classifier produced result fast, so it uses in most of the real time
systems.
B)Predictions of multi-class output - Naive bayes classifier algorithm predicts the posterior probability of
multiple classes of an input dataset.
C)Text Classification
D)Spam Classification
E)Recommendation system
3

4. ML- Unit-III Classification Department of IT
Types Of Naive Bayes
•There are three types of Naive Bayes model under the scikit-learn
library:
•Gaussian: It is used in classification and it assumes that features follow a
normal distribution.
•Multinomial: It is used for discrete counts. For example, let’s say, we have
a text classification problem. Here we can consider Bernoulli trials which is
one step further and instead of “word occurring in the document”, we have
“count how often word occurs in the document”, you can think of it as
“number of times outcome number x_i is observed over the n trials”.
•Bernoulli: The binomial model is useful if your feature vectors are binary
(i.e. zeros and ones). One application would be text classification with ‘bag
of words’ model where the 1s & 0s are “word occurs in the document” and
“word does not occur in the document” respectively.
4

5. Naive Bayes Classifier
Naive bayes classifier uses for binary or multiclass classification problems. It is known as naive Bayes
because the calculation of the probabilities for each hypothesis are simplified.
We will use following naive bayes classifier formula to calculate probability.
P(A|B) = P(B|A) . P(A)/P(B)
Where,
P(A|B) = It is a target probability of the class
P(B|A) = It is a probability of predictor given class
P(A) = It is the prior probability a class
P(B) = It is the prior probability of predictor
5

6. ML- Unit-III Classification Department of IT
Numerical Example
6

7. ML- Unit-III Classification Department of IT
Numerical Example
7
Solution:
P(A|B) = (P(B|A) * P(A) )/ P(B)
1.Mango:
P(X | Mango) = P(Yellow | Mango) * P(Sweet | Mango) * P(Long |
Mango)
2.a)P(Yellow | Mango) = (P(Yellow | Mango) * P(Yellow) )/ P
(Mango)
= ((350/650) * (800/1200)) / (650/1200)
P(Yellow | Mango)= 0.53 →1
1.b)P(Sweet | Mango) = (P(Sweet | Mango) * P(Sweet) )/ P
(Mango)
= ((450/650) * (850/1200)) / (650/1200)
P(Sweet | Mango)= 0.69 → 2

8. ML- Unit-III Classification Department of IT
Numerical Example
8
1.c)P(Long | Mango) = (P(Long | Mango) * P(Long) )/ P
(Mango)
= ((0/650) * (400/1200)) / (800/1200)
P(Long | Mango)= 0 → 3
On multiplying eq 1,2,3 ==> P(X | Mango) = 0.53 *
0.69 * 0
P(X | Mango) = 0

9. ML- Unit-III Classification Department of IT
Numerical Example
9
2. Banana:
P(X | Banana) = P(Yellow | Banana) * P(Sweet | Banana) * P(Long |
Banana)
2.a) P(Yellow | Banana) = (P( Banana | Yellow ) * P(Yellow) )/ P
(Banana)
= ((400/400) * (800/1200)) / (400/1200)
P(Yellow | Banana) = 2 → 4
2.b) P(Sweet | Banana) = (P( Banana | Sweet) * P(Sweet) )/ P
(Banana)
= ((300/400) * (850/1200)) / (400/1200)
P(Sweet | Banana) = 1.6 → 5

10. ML- Unit-III Classification Department of IT
Numerical Example
10
2.c)P(Long | Banana) = (P( Banana | Yellow ) * P(Long) )/ P
(Banana)
= ((350/400) * (400/1200)) / (400/1200)
P(Yellow | Banana) = 0.875 → 6
On multiplying eq 4,5,6 ==> P(X | Banana) = 2 * 1.6 * 0.875
P(X | Banana) = 2.8

11. ML- Unit-III Classification Department of IT
Numerical Example
11
3. Others:
P(X | Others) = P(Yellow | Others) * P(Sweet | Others) * P(Long |
Others)
3.a) P(Yellow | Others) = (P( Others| Yellow ) * P(Yellow) )/ P
(Others)
= ((50/150) * (800/1200)) / (150/1200)
P(Yellow | Others) = 1.78→ 7
3.b) P(Sweet | Others) = (P( Others| Sweet ) * P(Sweet) )/ P
(Others)
= ((100/150) * (850/1200)) / (150/1200)
P(Sweet | Others) = 3.78 → 8

12. ML- Unit-III Classification Department of IT
Numerical Example
12
3.c) P(Long | Others) = (P( Others| Long) *
P(Long) )/ P (Others)
= ((50/150) * (400/1200)) / (150/1200)
P(Long | Others) = 0.9 → 9
On multiplying eq 7,8,9 ==> P(X | Others) =
1.78 * 3.78* 0.9
P(X | Others) = 6.05

13. ML- Unit-III Classification Department of IT
Numerical Example
13
So finally from P(X | Mango) == 0 , P(X | Banana) ==
0.65 and P(X| Others) == 0.072.
We can conclude Fruit{Yellow,Sweet,Long} is
Banana.

14. Naive Bayes Classifier
Let we understand naive bayes classification with the help of solved example.
14
http://shatterline.com/blog/2013/09/12/not-so-naive-classification-with-the-naive-bayes-classifier/

15. Department of Info Tech
Naive Bayes Classifier
Ques - Consider we have 14 days data mentioned in earlier table. Find the probability to play a game on 15th
day where conditions are, temperature=cool, humidity= high, wind=strong, and outlook=sunny.
Answer - To predict the output based on input variables we have to follow following steps.
Step 1 - The last column mentioned in database is an output which depends upon the input variables.
Now, consider last column it consists of only 2 type of records (yes, no).
Total record (14)
This is division of last column according to the mentioned values.
Probability = P(Yes) = 9/14 P(No)= 5/14
15
Yes (9) No (5)

16. Naive Bayes Classifier
Step 2 - In this step consider the second last column from right hand side for processing. In this example we
consider wind column for processing. This column content 2 values i.e. weak and strong. Let we separate the
values of this column.
Wind (14)
Probability = P(Weak) = 8/14 P(strong)=6/14
16
weak(8) strong(6)
yes(6)
yes(3)no(2)
no(3)

17. Naive Bayes Classifier
Probability(weak|yes) = (6/9) Probability(strong|yes) = (3/9)
Probability(weak|no) = (2/5) Probability(strong|no) = (3/5)
17
weak(8) strong(6)
yes(6)
yes(3)no(2)
no(3)

18. Naive Bayes Classifier
Now step 2 calculations we have to repeat for next column i.e. Humidity
Probability(High|yes) = (3/9) Probability(Normal|yes) = (6/9)
Probability(High|no) = (4/5) Probability(Normal|no) = (1/5)
18
Humidity(14)
High(7) Normal(7)
yes(3)
no(4)yes(6) no(1)

19. Naive Bayes Classifier
Step 2 repeat for new column temperature.
19
Temperature(14)
Hot(4) mild(6) cool(4)
yes(2) yes(4) yes(3)
no(2) no(2) no(1)

20. Department of Info Tech
Naive Bayes Classifier
Probability(Hot|yes) = (2/9)Probability(mild|yes) = (4/9) Probability(cool|yes) = (3/9)
Probability(Hot|no) = (2/5)Probability(mild|no) = (2/5) Probability(cool|no) = (1/5)

20
Temperature(14)
Hot(4) mild(6) cool(4)
yes(2) yes(4) yes(3)
no(2) no(2) no(1)

21. MLA – Unit-V: Part-V Dr. Mahendra B. Gawali Department of Info Tech
Naive Bayes Classifier
Step 2 repeat for new column outlook.
21
outlook(14)
sunny(5)
overcast(4) rain(5)
yes(1)
yes(4)
yes(3)
no(4)
no(0) no(2)

22. Naive Bayes Classifier
Probability(sunny|yes) = (1/9) Probability(overcast|yes) = (4/9) Probability(rain|yes) = (3/9)
Probability(sunny|no) = (1/5) Probability(overcast|no) = (0/5)
Probability(rain|no) = (2/5)

22
outlook(1 4)
sunny(5)
overcast(4) rain(5)
yes(1)
yes(4)
yes(3)
no(4)
no(0) no(2)

23. Naive Bayes Classifier
Let A = {sunny, cool, high, strong}
P(A|yes) = P(yes) x P(sunny|yes) x P(cool|yes) x P(high|yes) x P(strong|yes)
Put all earlier calculated values in above equation
P(A|yes) = (9/14) x (1/9) x (3/9) x (3/9) x (3/9) = 0.0053
P(A|No) = P(No) x P(sunny|No) x P(cool|No) x P(high|No) x P(strong|No)
P(A|No) = 5/14 x x x x =
⅘ ⅕ ⅘ ⅗
0.0206
In last step compare both values
P(A|yes) = P (A|No)
0.0053 < 0.0206
The conclusion is on 15th day the game will not be played.
23