Cours setting up your Machine learning 2M2.pdf

Cours setting up your Machine learning version 2

1. Optimization
Algorithms
Mini-batch
gradient descent
deeplearning.ai

2. Andrew Ng
Batch vs. mini-batch gradient descent
Vectorization allows you to efficiently compute on mexamples.

3. Andrew Ng
Mini-batch gradient descent

4. Optimization
Algorithms
Understanding
mini-batch
gradient descent
deeplearning.ai

5. Andrew Ng
Training with mini batch gradient descent
# iterations
cost
Batch gradient descent
mini batch # (t)
cost
Mini-batch gradient descent

6. Andrew Ng
Choosing your mini-batch size

7. Andrew Ng
Choosing your mini-batch size

9. Andrew Ng

10. Andrew Ng

11. Optimization
Algorithms
Understanding
exponentially
weighted averages
deeplearning.ai

12. Andrew Ng
Exponentially weighted averages
days
temperature
!"=$!"%&+(1−$),"

13. Andrew Ng
Exponentially weighted averages
!"##=0.9!((+0.1+"##
!((=0.9!(,+0.1+((
!(,=0.9!(-+0.1+(,
…
!/=0!/1"+(1−0)+/

14. Andrew Ng
Implementing exponentially weighted
averages
!"=0
!%=&!"+(1−&) -%
…
!/=&!%+(1−&) -/
!0=&!/+(1−&) -0

15. Optimization
Algorithms
Bias correction
in exponentially
weighted average
deeplearning.ai

16. Andrew Ng
Bias correction
days
temperature
!"=$!"%&+(1−$),"

17. Optimization
Algorithms
Gradient descent
with momentum
deeplearning.ai

18. Andrew Ng
Gradient descent example

19. Andrew Ng
Implementation details
!"#=%!"#+1−%)*
!"+=%!"++1−%),
*=*−-!"#,
Hyperparameters: -,%
On iteration 8:
Compute)*,), on the current mini-batch
,=,−-!"+
%=0.9

20. Optimization
Algorithms
RMSprop
deeplearning.ai

21. Andrew Ng
RMSprop

22. Optimization
Algorithms
Adam optimization
algorithm
deeplearning.ai

23. Andrew Ng
Adam optimization algorithm
yhat= np.array([.9, 0.2, 0.1, .4, .9])

24. Andrew Ng
Hyperparameters choice:
Adam Coates

25. Optimization
Algorithms
Learning rate
decay
deeplearning.ai

26. Andrew Ng
Learning rate decay

27. Andrew Ng
Learning rate decay

28. Andrew Ng
Other learning rate decay methods

29. Optimization
Algorithms
The problem of
local optima
deeplearning.ai

30. Andrew Ng
Local optima in neural networks

31. Andrew Ng
Problem of plateaus
•Unlikely to get stuck in a bad local optima
•Plateaus can make learning slow