Reinforcement Learning with Human Feedback ODSC talk

Reinforcement Learning with Human Feedback, PPO, and DPO

1. Reinforcement Learning with Human Feedback
(RLHF)
Luis Serrano

2. Topics
•Large Language Models
•How to fine-tune them with reinforcement learning
•Quick intro to reinforcement learning
•PPO (the reinforcement learning technique to fine-tune LLMs)
•DPO (the non-reinforcement learning technique to fine-tune LLMs)

3. Serrano.Academy Serrano.Academy
Large Language Models

4. Serrano.Academy Serrano.Academy
Transformers
I generate text… one word at a time
Transformers

5. Serrano.Academy Serrano.Academy
Transformers
Hello, how are you doing
Transformers

6. Serrano.Academy Serrano.Academy
Transformers
Write a story. Once
Transformers

7. Serrano.Academy Serrano.Academy
Transformers
Write a story.Once upon
Transformers

8. Serrano.Academy Serrano.Academy
Transformers
Write a story.Onceupon a
Transformers

9. Serrano.Academy Serrano.Academy
How to train transformer models?
The internetCurated datasets
Transformers

10. Serrano.Academy Serrano.Academy
Is that the end of the story? (Nope)

11. Serrano.Academy Serrano.Academy
Transformers
Blue
What color is the sky?
Red
Banana
Transformers need to be fine tuned
by humans!

12. Serrano.Academy Serrano.Academy
Transformers need to be fine tuned
Transformers
It’s, ehhm… Cairo?
What is the capital of Sudan?
No! It’s
Khartoum!
by humans!

13. Quick intro to Reinforcement Learning

14. 1
1 1 1 3
+4
Gridworld
+4
+5
-1

15. 1
States

16. +4
+5
-1
1
1 1 1 3
Actions
+5
+4
1
1 1 1 3
-1

17. +4
+5
-1
1
1
Actions
+5
+4
1
1 1 1 3
-1

18. +5
+4 1
-1
It costs 1 to move one spot
4

19. +5
+4 1
-1
What’s the best I can do from here?
32
4
2

20. +5
+4 1
-1
What’s the best I can do from here?
32
4
1
0
-1
-4 -3 -2
3
2
2

21. +5
+4 1
-1
How to calculate all the values?
2
2
2
4
4
3
3
3
2
2
2
1
1
1
1
1
0
0
0
-2
-2-1
-1
-1
3
3
-2
-2

22. Policy

23. Policy if it’s free to move
+5
+4
1
1 1 1 3
-1

24. +5
+4 1
-1
Where is the best place to move?
2 1 2 3 4
3 2 1 2 3 4
0 1 2 3
3 -1
2 1 0 -1 -2 -2
1 0 -1 -2 -2

25. +5
+4 1
-1
Stochastic Policy
4-3
1
2

26. Neural networks

27. Problem solved! (Or is it?)
+5
+4
1
1 1 1 3
-1
2 1 2 3 4+5
3 2 1 2 3 4
+4
1 0 1 2 3
3 1 -1 1 1 3
2 1 0 -1 -2 -2
1 0 -1 -2 -2-1
Values Policy

28. Value network
x
y
Value
Neural
network
x
y
score = 3
x
y
3

29. Policy network
x
y
x
y
p( )
p( )
p( )
p( )
x
y
= 0.60
= 0.25
= 0.05
= 0.10
Policy
Neural
network

30. Value neural network
2 1 2 3 4
+5
3 2 1 2 3 4
+4
1 0 1 2 3
3 1 -1 1 1 3
2 1 0 -1 -2 -2
1 0 -1 -2 -2
-1
Values
3.11.42.53.22.7
+5
2.2-0.21.32.94.13.2
+4
1 -0.51.72.83.2
3.9 1 0.2 1 1 3
2.61.51.10.3-1.8-2.3
1.40.80.1-1.5-2
-1
Value neural network

31. Policy neural network
Policy Policy neural network
+5
+4
1
1 1 1 3
-1
+5
+4
1
1 1 1 3
-1
Deterministic (exploit) Stochastic (explore-exploit)

32. Serrano.Academy Serrano.Academy
RLHF

33. Serrano.Academy Serrano.Academy
Blue
What color is the sky?
Red
Banana
Transformers need to be fine tuned
by humans!

34. Serrano.Academy Serrano.Academy
Training the value network
What color
is the sky?
What color
is the sky?
Blue
What color
is the sky?
Red
What color
is the sky?
BananaBlue Banana
Red
+2
+3
+1
Value
neural
network
Blue
Red
Banana
Policy
neural
network

35. Serrano.Academy Serrano.Academy
Roses are red
Longer paths
+2
+3
+1
Roses are red,
because they like
ketchup.
Roses are red,
violets are blue
Roses are red,
violets are humans
too
Roses are red,
violets are
humans
Roses are red,
violets are
humans too
Roses are red,
violets are
humans too
<end>
Roses are red,
Roses are red,
violets
Roses are red,
violets are
Roses are red,
because they
like
Roses are red,
because they
like ketchup
Roses are red,
because they
like ketchup
<end>
Roses are red,
because they
Roses are red,
because
Roses are red,Roses are red,
Roses are red,
violets
Roses are red,
violets are
Roses are red,
violets are
blue
Roses are red,
violets are
blue <end>
Value
neural
network
Policy
neural
network

36. Serrano.Academy Serrano.Academy
Obviously it’s not a grid
What color
is the sky?
Hello, how
are you?
The Iliad
Once upon a
time…
Hello, how
are you?
Good

37. Serrano.Academy Serrano.Academy
Obviously it’s not a grid
What color
is the sky?
What color
is the sky?
The sky is
blue
What color
is the sky?
The sky is
What color
is the sky?
The sky
What color
is the sky?
The
What color
is the sky?
Banana
monkeyWhat color
is the sky?
Banana
What color
is the sky?
The
What color
is the sky?
Banana
monkey
dishwasher
Hello, how
are you?
The Iliad
Once upon a
time…
What color
is the sky?
The sky is
blue
What color
is the sky?
The sky is
What color
is the sky?
The sky
What color
is the sky?
The
What color
is the sky?
Banana
purple
monkey
What color
is the sky?
Banana
purple
What color
is the sky?
Banana
What color
is the sky?
Banana
purple
monkey
dishwasher
Hello, how
are you?
Good

38. Serrano.Academy Serrano.Academy
Obviously it’s not a grid
What color
is the sky?
What color
is the sky?
The sky is
blue
What color
is the sky?
The sky is
What color
is the sky?
The sky
What color
is the sky?
The
What color
is the sky?
Banana
monkeyWhat color
is the sky?
Banana
What color
is the sky?
The
What color
is the sky?
Banana
monkey
dishwasher
What color
is the sky?
The sky is
blue
What color
is the sky?
The sky is
What color
is the sky?
The sky
What color
is the sky?
The
What color
is the sky?
Banana
purple
monkey
What color
is the sky?
Banana
purple
What color
is the sky?
Banana
What color
is the sky?
Banana
purple
monkey
dishwasher
Value
neural
network
Policy
neural
network
+3
+1

39. Proximal Policy Optimization
How to train the two networks:

40. Training the value neural network

41. Two neural networks
+5
+41
1 1 1 3
-1
2 1 2 3 4+5
3 2 1 2 3 4
+41 0 1 2 3
3 1-11 1 3
2 1 0-1-2-2
1 0-1-2-2-1
Value
neural
network
Policy
neural
network
Approximates
values
Approximates
policy

42. Loss function for the training the value network
L
value
(θ)=??????[(V
θ(s
t)−R
t)
2
]
2 1 2 3 4+5
3 2 1 2 3 4
+41 0 1 2 3
3 1-11 1 3
2 1 0-1-2-2
1 0-1-2-2-1

43. +5
+4
1
1 1 1 3
-1
Training the value neural network
1.2-10.50.60.8+5
0.81.2-0.50.40.51.2
+4
1-0.30.61.20.4
-0.3 10.8 1 1 3
-10.5-1-0.30.40.5
0.800.50.4-0.3-1
Value neural network Actual values
0.40.51.2
-0.30.6
0.8
32
10
-1
42 3 4
0 1
-1
Policy neural network

44. Training the value neural network
Value neural network
2 3 4
0 1
-1
0.40.51.2
-0.30.6
0.8
Actual values
−
2.566.257.84
0.090.16
3.24
=
( )
2
1
6
7.846.252.56
0.160.09
3.24

45. 0000000000000000000000000000
6
Training the value neural network
Value neural network
2 3 4
0 1
-1
0.40.51.2
-0.30.6
0.8
Actual values
−
=
( )
2
Loss
1
6
7.846.252.560.160.093.24+ + + + +
=3.36

46. 0000000000000000000000000000
6
Training the value neural network
=
Loss
7.846.252.560.160.093.24+ + + + +
=3.36
Use this loss to
train the value
neural network

47. Training the value neural network
1.2-10.50.60.8+5
0.81.2-0.50.40.51.2
+4
1-0.30.61.20.4
-0.3 10.8 1 1 3
-10.5-1-0.30.40.5
0.800.50.4-0.3-1
0.60.91.8
-0.20.7
0.6
Value neural network
0.1
-0.6
1.40.50.4
0.40.1-0.3
-0.7
0.50.8
2.11.7
3.11.42.53.22.7+5
2.2-0.21.32.94.13.2
+4
1 -0.51.72.83.2
3.9 1 0.2 1 1 3
2.61.51.10.3-1.8-2.3
1.40.80.1-1.5-2-1

48. Formula for the Loss function
L
value
(θ)=??????[(V
θ(s
t)−R
t)
2
]
Expected value (average)
2 3 4
0 1
-1
0.40.51.2
-0.30.6
0.8
−( )
2
1
6

49. Formula for the Gain
L
value
(θ)=??????[(V
θ(s
t)−R
t)
2
]
Values given by the neural network
2 3 4
0 1
-1
0.40.51.2
-0.30.6
0.8
−( )
2
1
6

50. Formula for the Gain
L
value
(θ)=??????[(V
θ(s
t)−R
t)
2
]
Actual values (calculated using the path)
2 3 4
0 1
-1
0.40.51.2
-0.30.6
0.8
−( )
2
1
6

51. Formula for the Gain
L
value
(θ)=??????[(V
θ(s
t)−R
t)
2
]
Actual values (calculated using the path)
2 3 4
0 1
-1
0.40.51.2
-0.30.6
0.8
−( )
2
1
6

52. Formula for the Gain
L
value
(θ)=??????[(V
θ(s
t)−R
t)
2
]
Use this loss to
train the value
neural network

53. Training the policy neural network: The surrogate

54. Two neural networks
2 1 2 3 4+5
3 2 1 2 3 4
+41 0 1 2 3
3 1-11 1 3
2 1 0-1-2-2
1 0-1-2-2-1
Value
neural
network
Perfect
values
+5
+41
1 1 1 3
-1
Policy
neural
network
Perfect
policy

55. Loss function for training the policy network
L
policy
(θ)=
π
θ
(a
t
|s
t
)
π
θ
old
(a
t|s
t)
A
t
+5
+41
1 1 1 3
-1

56. +5
+4
1
1 1 1 3
-1
Training the value neural network
1.2-10.50.60.8+5
0.81.20.80.40.51.2
+4
1 11.31.20.4
-0.3 10.8 1 1 3
-10.5-1-0.30.40.5
0.800.50.4-0.3-1
Value neural network Actual values
0.40.51.2
11.3
0.8
32
10
-1
42 3 4
0 1
-1
Policy neural network

57. +5 +5
Training the value neural network
Value neural network Actual values
0.40.51.2
11.3
0.8
32
10
-1
41.2 4
−p()( )
Gain
π
θ
(a
t
|s
t
)
π
θ
old
(a
t|s
t)
A
t
Impulse

58. Impulse (Momentum)
v = 10 v = 20 v = -10 v = 5
Force Force
Scenario 1 Scenario 2
Impulse Impulse

59. Impulse (Momentum)
Iteration
Probability
Gain
Gain
Scenario 1 Scenario 2
Iteration
Probability
Impulse Impulse

60. Impulse (Momentum)
Iteration
Probability
Gain
Gain
Scenario 3 Scenario 4
Iteration
Probability
Impulse
Impulse

61. Impulse (Momentum)
Iteration 99
p=0.2
Iteration 100 10
p=0.4
Iteration 101
p=0.8
Gain
Scenario 1 Scenario 2
Iteration 99
p=0.8
Iteration 100 10
p=0.4
Iteration 101
p=0.5
Gain

62. 10 10
Dividing by the previous probability
Iteration 99
p=0.2
Iteration 100 10
p=0.4 Gain
Scenario 1 Scenario 2
Iteration 99
p=0.8
Iteration 100 10
p=0.4 Gain
0.4
0.2
Surrogate
objective
function
0.4
0.8
=20 =5
0.4
0.2
0.4
0.8
10 10

63. Surrogate Objective Function
L
policy
(θ)=
π
θ(a
t
|s
t)
π
θ
old
(a
t|s
t)
A
t
Probability at iteration 100
Probability at iteration 99
Gain (difference of value)
at iteration 100

64. Serrano.Academy Serrano.Academy
Direct Preference Optimization

65. Serrano.Academy Serrano.Academy

66. Serrano.Academy Serrano.Academy

67. Serrano.Academy Serrano.Academy
Score: 2
Score: 5
Reinforcement Learning with Human Feedback (RLHF)
banana
time
Transformer
Once upon a
Reward: 2
Reward: 5
Reward
Model
Transformer
Model
(the one that talks)
(the one that guesses how
the human will rate outputs)
Direct Preference Optimization (DPO)
Loss function
ℒ
DPO
(π
θ
;π
ref
)=−??????
(x,y
w,y
l)∼D
logσ
(
βlog
π
θ
(y
w
|x)
π
ref
(y
w|x)
−βlog
π
θ
(y
l
|x)
π
ref
(y
l|x))

68. Serrano.Academy Serrano.Academy
Score: 2
Score: 5
banana
time
Transformer
Once upon a
Reward: 2
Reward: 5
Reward
Model
Transformer
Model
(the one that talks)
(the one that guesses how
the human will rate outputs)
Direct Preference Optimization (DPO)
max
π
θ
??????
x∼??????,y∼π
θ(y|x)
[r
ϕ
(x,y)]−β??????
KL
[π
θ
(y|x)∥π
ref
(y|x)]πr π

69. Serrano.Academy Serrano.Academy
DPO main equation
max
π
θ
??????
x∼??????,y∼π
θ(y|x)
[r
ϕ
(x,y)]−β??????
KL
[π
θ
(y|x)∥π
ref
(y|x)]
Maximizes the rewards Prevents the model from
changing too drastically
πr π
x=Once upon a y=time
xy y yx x
y=banana

70. Serrano.Academy Serrano.Academy
Two things
max
π
θ
??????
x∼??????,y∼π
θ(y|x)
[r
ϕ
(x,y)]−β??????
KL
[π
θ
(y|x)∥π
ref
(y|x)]πr πxy y yx x
How to get rid of this reward?
(not get rid of it, but turn it into a probability)
How to measure how different two models are?
KL Divergence
Bradley-Terry Model
Original
model
Improved
model

71. Serrano.Academy Serrano.Academy
Turning rewards(scores) into probabilities
max
π
θ
??????
x∼??????,y∼π
θ(y|x)
[r
ϕ
(x,y)]rxy
Bradley-Terry Model
How to get rid of this reward?
(not get rid of it, but turn it into a probability)
σ(r
w
−r
l
)

72. Serrano.Academy Serrano.Academy
Mathematical Manipulation
The DPO loss function
ℒ
DPO
(π
θ
;π
ref
)=−??????
(x,y
w,y
l)∼D
logσ
(
βlog
π
θ
(y
w
|x)
π
ref
(y
w|x)
−βlog
π
θ
(y
l
|x)
π
ref
(y
l|x))
p(y
1
>y
2
)=
e
r(x,y
1)
e
r(x,y
1)
+e
r(x,y
2)
Bradley Terry Model
max
π
θ
??????
x∼??????,y∼π
θ(y|x)
[r
ϕ
(x,y)]−β??????
KL
[π
θ
(y|x)∥π
ref
(y|x)]πr π

73. Serrano.Academy Serrano.Academy
The DPO loss function
ℒ
DPO
(π
θ
;π
ref
)=−??????
(x,y
w,y
l)∼D
logσ
(
βlog
π
θ
(y
w
|x)
π
ref
(y
w|x)
−βlog
π
θ
(y
l
|x)
π
ref
(y
l|x))
log
∏
p
i
Maximize probability
for good response
Don’t change the
model too much
Average over
generated responses
∑
logp
i=
Probability
of some event
Probability
of all events
(independently)
Minimize probability
for bad response
max
min

74. Serrano.Academy Serrano.Academy

75. Serrano.Academy Serrano.Academy
Series of 4 videos in Reinforcement Learning

76. Serrano.Academy Serrano.Academy
Grokking Machine Learning

77. Serrano.Academy Serrano.Academy
llm.university