The Pepsi Challenge: A Statistical Problem in Tasting Studies

Pepsi Challenge

1. The Pepsi Challenge
A Statistical Problem in
Tasting Studies
Dr Lee Fawcett

2. Comparing different
products
• Who are (hopefully) typical of the
• Tasting studies consist of
judges
target
population

3. Simplest
situation
• Two forms of a product are to be compared:
• Each judge tastes both A and B and either
chooses the “odd one out” or chooses which
they prefer
• Question: Can people tell the difference? If
they can, do people have a preference?

4. Possible
solutions?
• Make sure that A and B cannot be distinguished
by anything other than taste
• Randomise the order of tasting so that there is
an equal chance of tasting A and B first
(sometimes the first product is preferred)
• Use each product more than once, which will
allow us to assess whether:
– people give consistent answers
– people really can tell the difference/have a
preference

5. An example: Triangle
tests
Each judge will receive:
or
Judges are asked to choose the odd one out

6. The Pepsi
Challenge!
You will sample:
Two cups of A and one cup of B
or
One cup of A and two cups of B
Each judge has a number (1,2,3,…) and each
cup has a letter (X,Y,Z) denoting which person
and cup is being sampled.

7. Statistical Analysis
I
We can test the hypothesis:
H
0
: People cannot distinguish between A and B
• If people cannot distinguish we expect 1/3 to
choose the correct ‘odd one out’ by chance
• As people choose independently, the number who are
correct should follow a Binomial distribution with p =
1/3:

8. Statistical Analysis
II
H
0
: People cannot distinguish between A and B

9. Suppose we have 30 judges…

10. If people can tell the
difference, we’ll get a result
up here somewhere

11. Let’s say 16
people out of our
30 correctly
identified the odd
one out

12. Is what we’ve
observed far enough
away from what we’d
expect to dismiss H
0
?

13. We calculate the
probability of what
we’ve observed
and anything more
extreme than this.
This is the p-value

14. Experimental Design
I
X Y Z
A A B
A B A
B A A
B B A
B A B
A B B
Building blocks:
Randomise the line order (4 1 6 3 5 2) and
repeat to give………

15. Experimental Design II
JUDGE X Y Z
1 B B A
2 A A B
3 A B B
4 B A A
5 B A B
6 A B A
7 B B A
8 A A B
9 A B B
10 B A A
11 B A B
12 A B A
13 B B A
14 A A B
15 A B B
16 B A A
17 B A B
18 A B A
. . . .
. . . .