Lecture 4 Supervised Learning part 3.pptx

Machine Learning

1. Supervised Learning (Part 3) Md. Shahidul Islam Assistant Professor Dept. of CSE, University of Asia Pacific

2. Gradient The gradient of a scalar function is a vector that points in the direction of the greatest rate of increase of the function. The gradient of a function tells us two things: Direction of Steepest Ascent: the direction in which the function increases the fastest. Rate of Increase: the magnitude of the gradient tells how fast the function increases in that direction. 2

3. Gradient Descent Intuition : gradient tells us direction of greatest increase, negative gradient gives us direction of greatest decrease If we take a small enough step in the direction of the negative gradient, the function will decrease in value Goal : to minimize MSE Take the negative gradient to update parameter that reduces the error 3

4. Gradient of the function   Path taken by gradient descent, starting from the point [3.0,3.0] 4

5. Gradient Descent Algorithm: Pick an initial point Gradient Descent Update Rule Where is the current position at iteration is the step size (learning rate) is the gradient (first derivative) of the function Iterate until convergence moves in the negative direction of the gradient to minimize   5

6. Gradient Descent When to stop? Iterate until for some A typical choice of threshold, ϵ=10 −6 How to choose step size ? Try small values first (e.g., 0.01, 0.1, 0.5). If gradient descent diverges (i.e., 𝑓 ( 𝑥 ) keeps increasing), decrease 𝛼. If convergence is too slow, try increasing 𝛼. Instead of using fixed value, try using Line Search to find the best step size.   6

7. Gradient Descent f(x) = x 2 nitial x = -4 Step size, = 0.8   7

8. Gradient Descent 8

9. Gradient Descent 9

10. Gradient Descent 10

11. Gradient Descent 11

12. Gradient Descent 12

13. Gradient Descent for Linear Regression Cost function: Update weight using Gradient Descent.   13

14. Fitting a Regression Line Starting with w = 0, b = 0, = 0.2   House Size (Normalized) Rent ($) 0.0 1.5 0.4 2.0 0.7 2.5 1.0 3.0 14

15. Logistic regression is used for classification!! Predicts the probability of a binary outcome (0 or 1) Extends the idea of a linear regression to classification tasks Logistic Regression 15

16. Logistic Function 17

17. Linear Regression vs Logistic Regression Key Differences: Linear Regression : Predicts continuous outcomes, assumes a straight-line relationship between the independent and dependent variables. Polynomial Regression : Extends linear regression to model non-linear relationships by using polynomial terms of the independent variable. Logistic Regression : Predicts probabilities for binary outcomes, using the logistic function (sigmoid) to squeeze the output between 0 and 1. It is used for classification rather than regression. 18

18. Questions? 19