slides_security_and_privacy_in_machine_learning.pptx

.

1. N i co l a s P a p e r n ot P e nn s y l v a nia S t a t e U ni v e r s i t y & G oogle B r a in L ec t ure for P rof. T r e n t J a e g e r ’ s C SE 54 3 C ompu t e r Se cur i t y C l a s s N ov e mb e r 201 7 - P en n S t a t e

2. ( P e n n S tate) ( Googl e Bra in ) M a r t ín Ab ad i ( Goog l e B r a in ) Som e s h J h a ( U of W i s con s in ) T h a n k you t o my co ll a bor a t ors 2 A l e x e y K ur a k i n ( Goog l e B r a in ) X i Wu ( Goog l e )

3. 3 M a c hin e L ea r nin g Classi f ie r . 01 . 84 . 02 . 01 . 03 . 01 p(0 | x , θ) p(1 | x , θ) p(2 | x , θ) p(7 | x , θ) p(8 | x , θ) p(9 | x , θ)

4. M a c h in e L e a r n in g C l a ss i f i e r cos t/ loss f unc t ion ( ~ model error) 4

5. O u t lin e of t hi s le c t ure 1 2 5

6. P a rt I S e cur i t y i n m a c h in e le a r n in g 6

7. b a d eve n i f a n a tt a c k e r need s t o k n ow de t a il s of t he m a c h in e le a r n in g mo d e l t o d o a n a tt a ck -- - a k a a w h i t e - b ox a tt a c k er wor s e i f a tt a c k e r w h o k n ows ve ry li tt l e ( e .g. o n l y g e t s t o a s k a f e w qu e s t i o n s ) c a n d o a n a tt a ck -- - a k a a b l a c k - b ox a tt a c k er ML A t t a ck Mo d el s 7

8. A t t a ck Mo d el s b a d eve n i f a n a tt a c k e r need s t o k n ow de t a il s of t he m a c h in e le a r n in g mo d e l t o d o a n a tt a ck -- - a k a a w h i t e - b ox a tt a c k er wor s e i f a tt a c k e r w h o k n ows ve ry li tt l e ( e .g. o n l y g e t s t o a s k a f e w qu e s t i o n s ) c a n d o a n a tt a ck -- - a k a a b l a c k - b ox a tt a c k er ML 8

9. A d v e r s a r i a l e x a mp l e s ( w h i t e - box a tt a c k s ) 9

10. J a cob i a n - b a s e d S a lien cy M a p A ppro a ch ( J SMA) a t ions of Deep Learning in A dversarial S e tt ings 1 P apernot et al. T he Limit

11. J a cob i a n - B a s e d I t e ra t i ve Appro a c h : s o u r ce - t a rg e t m i s c l a ss i f i c a t ion 1 1 P apernot et al. T he Limi t a t ions of Deep Learning in A dversarial S e tt ings

12. Ev a din g a N e ur a l N e t work M a l w a re C l a ss i f i e r P [ X = Beni g n ] = . 1 P [ X * = B e ni g n ] = . 9 1 2 G rosse et al. A dversarial P er t urba t ions A gainst Deep Neural Ne t works f or Malware Classi f ica t ion

13. Sup e rv i s e d v s . r e in forc e m e n t le a r n in g 1 3 Sup erv i sed l ear nin g Re in f o rceme n t l ear nin g M od el inpu ts O bserva t ion (e . g . , t ra ff ic sign, music, email) E nvironment & Reward f unc t ion M od el ou t pu ts Class (e . g . , s t op / yield, jazz / classical, spam / legi t ima t e) A c t ion T ra inin g “ go a l ” ( i. e . , c o st /lo ss) Minimize class predic t ion error over pairs of (inpu t s, ou t pu t s) Maximize reward by exploring t he environment and t aking ac t ions E xam pl e

14. A d v e r s a r i a l a tt a c k s on ne ur a l ne t work po l i c i e s 1 4 Huang et al. A dversarial Att acks on Neural Ne t work P olicies

15. A d v e r s a r i a l e x a mp l e s ( b l a c k - box a tt a c k s ) 15

16. T h r e a t mo d e l of a b l a c k - box a tt a ck 16

17. O ur a ppro a ch t o b l a c k - box a tt a c k s A llevia t e lack of knowledge about model 17 A llevia t e lack of t raining da t a

18. A d v e r s a r i a l e x a mp l e t ra n s f e ra b i li t y T he s e prop e r t y com e s i n s e v e ra l v I n t ra - t e c h n i que t ra n s f e ra b i li t y: a r i a n t s : C ro s s mo d e l t ra n s f e ra b ili t y C ro s s t ra i n i n g s e t t ra n s f e ra b ili t y C ro s s - t e c h n i que t ra n s f e ra b i li t y ML A 18 S zegedy et al. I n t riguing proper t ies of neural ne t works

19. T he s e prop e r t y com e s i n s e v e ra l v I n t ra - t e c h n i que t ra n s f e ra b i li t y: a r i a n t s : C ro s s mo d e l t ra n s f e ra b ili t y C ro s s t ra i n i n g s e t t ra n s f e ra b ili t y C ro s s - t e c h n i que t ra n s f e ra bi li t y ML A ML B Vi ct i m A d v e r s a r i a l e x a mp l e t ra n s f e ra b i li t y 19 S zegedy et al. I n t riguing proper t ies of neural ne t works

20. 2 A d v e r s a r i a l e x a mp l e t ra n s f e ra b i li t y

21. C ro s s - t e c h ni que t ra n s f e ra b i li t y 21 P apernot et al. T rans f erabili t y in Machine Learning: f rom P henomena t o B lack- B ox Att acks using A dversarial S amples

22. C ro s s - t e c h ni que t ra n s f e ra b i li t y 22 P apernot et al. T rans f erabili t y in Machine Learning: f rom P henomena t o B lack- B ox Att acks using A dversarial S amples

23. O ur a ppro a ch t o b l a c k - box a tt a c k s 23 A llevia t e lack of t raining da t a A llevia t e lack of knowledge about model

24. A t t a c k in g r e mo t el y h o s t e d b l a c k - box mo d el s Rem o te ML sys 24

25. Rem o te ML sys A t t a c k in g r e mo t el y h o s t e d b l a c k - box mo d el s 25

26. 2 Rem o te ML sys A t t a c k in g r e mo t el y h o s t e d b l a c k - box mo d el s 6

27. Rem o te ML sys A t t a c k in g r e mo t el y h o s t e d b l a c k - box mo d el s 2 7

28. O ur a ppro a ch t o b l a c k - box a tt a c k s 2 8 A llevia t e lack of knowledge about model A llevia t e lack of t raining da t a

29. R e s u l t s on r e a l - wor l d r e mo t e s y s t e ms 2 9 [P M G 16a] P apernot et al. P rac t ical B lack- B ox Att acks against Deep Learning S ys t ems using A dversarial E xamples R em o t e Pla t fo r m M L te c h ni qu e N umbe r of quer i e s A dv er sa r ial ex a mp l e s m isclassifi e d (af te r query in g ) Dee p L ea r nin g 6 , 40 84 . 24 % Log isti c Re gr essi on 80 96 . 19 % U nkn own 2 , 00 97 . 72 %

30. Ben c h m a r k in g progre s s i n t h e a d v e r s a r i a l ML commu n i t y 3

31. 3 1

32. Grow i n g commu n i t y 1 . 3 K + s t a rs 340 + fork s 40 + con t r i bu t ors 32

33. 33

34. 3 4 A d v e r s a r i a l e x a mp l e s a re a t a ngible in s t a n ce of h ypo t he t i c a l AI s a f e t y prob l e ms I mage source: h tt p :// www . nerdis t. com / wp-con t en t/ uploads / 2013 / 07 /S pace- O dyssey-4 . jpg

35. P a rt I I P r i v a cy i n m a c h in e le a r n in g 35

36. T yp e s of a d v e r s a r i e s a n d our t h r e a t mo d e l I n o u r w o r k, t h e t h r e at m od e l ass u m e s: A d v e r s a ry c a n m a k e a po t e n t i a ll y u n bou n d e d n umb e r of qu e r i e s A d v e r s a ry h a s a cc e s s t o mo d e l i n t e r n a l s Mo d e l qu e ry i n g ( b lack- b ox adv e r sa r y ) B lack-box ML 36

37. A de f i ni t i on of pr i v a cy 37

38. O ur de s i gn go a l s 38

39. T h e P AT E a ppro a ch 39

40. T e ac h e r e ns e m b le Se n s i t i ve D a t a 40 Pa r titi on 1 Tea c he r 1 Pa r titi on 2 Tea c he r 2 Pa r titi on n Tea c he r n Pa r titi on 3 Tea c he r 3

41. A gg r e g a t ion 41

42. I n tu i t ive p r ivacy anal y sis 42

43. Noisy a g g r e g a t ion 43

44. T e ac h e r e ns e m b le P a r t i t i on 3 T e a c h e r 3 Aggr e g a t e d T e a c h e r Se n s i t i ve D a t a 44 Pa r titi on 1 Tea c he r 1 Pa r titi on 2 Tea c he r 2 Pa r titi on n Tea c he r n

45. S t u d e nt tr aining P a r t i t i on 3 T e a c h e r 3 Aggr e g a t e d T e a c h e r Se n s i t i ve D a t a 45 Pa r titi on 1 Tea c he r 1 Pa r titi on 2 Tea c he r 2 Pa r titi on n Tea c he r n S t u den t P ub li c Dat a

46. W h y tr ain an addi t ional “ s t u d e n t ” m od e l? 1 2 46

47. S t u d e nt tr aining P a r t i t i on 3 T e a c h e r 3 Aggr e g a t e d T e a c h e r Se n s i t i ve D a t a 47 Pa r titi on 1 Tea c he r 1 Pa r titi on 2 Tea c he r 2 Pa r titi on n Tea c he r n S t u den t P ub li c Dat a

48. D e p lo y m e nt 48 S t u den t

49. D iff e r e n t ial p r ivacy anal y sis 49

50. E x p e r i m e n t a l r e s u l t s 50

51. E x p e r i m e n t al s e t u p 5 1 Dataset T eac h er M od el S t ud e n t M od el MN IS T Convolu t ional Neural Ne t work G enera t ive A dversarial Ne t works SV HN Convolu t ional Neural Ne t work G enera t ive A dversarial Ne t works UCI A dul t Random F orest Random F orest UCI D i a b etes Random F orest Random F orest / /model s /tree/master/differential_privacy/multiple_teachers

52. A gg r e g a te d t e ac h e r acc u r acy 52

53. T r ad e - off b e t w een s t u d e nt acc u r acy and p r ivacy 53

54. T r ad e - off b e t w een s t u d e nt acc u r acy and p r ivacy 54 UCI D i a b etes ᷑ 1 . 44 ᶖ 1 -5 N on - p r i vate b ase lin e 93 . 81% S t ud e n t acc u racy 93 . 94%

55. Sy n e rgy b e t w e e n p r ivacy a n d g e n e r ali z a t ion 55

56. www.p a p er no t . fr @N i c ol as P a p er no t 56

57. 57

58. Gr a dien t m a s k in g 58

59. Gr a dien t m a s k in g 59