Cross-Regional Malware Detection via Model Distilling and Federated Learning

My paper at the RAID conference on how to distill ML-based malware detection models to detect malware all around the world.

1. Regional MalwareFL and Distillation Conclusion
Cross-Regional Malware Detection via Model Distilling and
Federated Learning
Marcus Botacin
1
1
Assistant Professor
Texas A&M University (TAMU), USA
[email protected]
@MarcusBotacin
Cross-Regional Malware Detection via Model Distilling and Federated Learning 1 / 52

2. Regional MalwareFL and Distillation Conclusion
Agenda
1
Regional Malware
The Dierences
2
FL and Distillation
New Architecture
Case Study
3
Conclusion
Generalization
Final Remarks
Cross-Regional Malware Detection via Model Distilling and Federated Learning 2 / 52

3. Regional MalwareFL and Distillation Conclusion
The Dierences
Agenda
1
Regional Malware
The Dierences
2
FL and Distillation
New Architecture
Case Study
3
Conclusion
Generalization
Final Remarks
Cross-Regional Malware Detection via Model Distilling and Federated Learning 3 / 52

4. Regional MalwareFL and Distillation Conclusion
The Dierences
Previously
Figure: Link:https://dl.acm.o
rg/doi/10.1145/3429741
Figure: Source:
https://www.us
enix.org/confe
rence/enigma20
21/presentatio
n/botacin
Figure: Source:
https://dl.acm.org/doi/1
0.1145/3339252.3340103
Cross-Regional Malware Detection via Model Distilling and Federated Learning 4 / 52

5. Regional MalwareFL and Distillation Conclusion
The Dierences
Impact on AV Detection
Figure: Source:
https://www.sciencedirect.com/scienc
e/article/pii/S01674048203013100.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
55.0%
60.0%
65.0%
70.0%
75.0%
80.0%
85.0%
90.0%
95.0%
100.0%
World (PE)Brazil (PE)World (Web)Brazil (Web)
Detection Rate (%)
Dataset
AV detection evolution after 30 days
Final
Initial
Figure: Detection Rate:BR samples are
consistently less detected.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 5 / 52

6. Regional MalwareFL and Distillation Conclusion
The Dierences
The Current Dataset
Table: Dataset Dierences.Dynamic analysis events for the US, Brazil, and Japan datasets.
Behavior US BR JP
Hosts le modication 0.04% 1.09% 0.92%
File creation 64% 24% 70%
File deletion 34% 12% 34%
File modication 63% 16% 46%
Browser modication 0% 1.03% 0.59%
Network trac 53% 96% 52%
Cross-Regional Malware Detection via Model Distilling and Federated Learning 6 / 52

7. Regional MalwareFL and Distillation Conclusion
The Dierences
The Traditional Architecture
Figure: Single Model
Distillation.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 7 / 52

8. Regional MalwareFL and Distillation Conclusion
The Dierences
Is it enough to have global models?
Cross-Regional Malware Detection via Model Distilling and Federated Learning 8 / 52

9. Regional MalwareFL and Distillation Conclusion
The Dierences
US Features: Accuracy 0 30 60 90 120 150 180 210 240 270
Features (#)
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Accuracy (%)
Classification Accuracy vs. Number of Features (US)
N=2
N=2000
Figure: Accuracy rates for the US dataset.Accuracy variation with the increase of the
feature set until reaching the 99% value.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 9 / 52

10. Regional MalwareFL and Distillation Conclusion
The Dierences
US Features: Size 0 30 60 90 120 150 180 210 240 270
Features (#)
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
4000000
4500000
5000000
5500000
Nodes (#)
Model Size vs. Number of Tree Nodes (US)
N=2
N=100
N=500
N=1000
N=2000
Figure: Model size for the US dataset.Number of nodes for an increased number of
ensemble trees of increasing feature set sizes.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 10 / 52

11. Regional MalwareFL and Distillation Conclusion
The Dierences
BR Features: Accuracy 0 30 60 90 120150180210240270300330
Features (#)
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Accuracy (%)
Classification Accuracy vs. Number of Features (BR)
N=3
N=2000
Figure: Accuracy rates for the BR dataset.Accuracy variation with the increase of the
feature set until reaching the 99% value.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 11 / 52

12. Regional MalwareFL and Distillation Conclusion
The Dierences
BR Features: Size 0 30 60 90120150180210240270300330
Features (#)
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
4000000
4500000
5000000
5500000
Nodes (#)
Model Size vs. Number of Tree Nodes (BR)
N=3
N=100
N=500
N=1000
N=2000
Figure: Model size for the BR dataset.Number of nodes for an increased number of
ensemble trees of increasing feature set sizes.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 12 / 52

13. Regional MalwareFL and Distillation Conclusion
The Dierences
JP Features: Accuracy 01002003004005006007008009001000110012001300
Features (#)
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Accuracy (%)
Classification Accuracy vs. Number of Features (JP)
N=2
N=2000
Figure: Accuracy rates for the JP dataset.Accuracy variation with the increase of the
feature set until reaching the 99% value.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 13 / 52

14. Regional MalwareFL and Distillation Conclusion
The Dierences
JP Features: Size 0 100 200 300 400 500 600 700 800
Features (#)
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
4000000
4500000
5000000
5500000
Nodes (#)
Model Size vs. Number of Tree Nodes (JP)
N=2
N=100
N=500
N=1000
N=2000
Figure: Model size for the JP dataset.Number of nodes for an increased number of
ensemble trees of increasing feature set sizes.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 14 / 52

15. Regional MalwareFL and Distillation Conclusion
The Dierences
Global Features: Accuracy 0 100 200 300 400 500 600 700
Features (#)
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Accuracy (%)
Classification Accuracy vs. Number of Features (Global)
N=3
N=2000
Figure: Accuracy rates for the combined dataset.Accuracy variation with the increase of
the feature set until reaching the 99% value.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 15 / 52

16. Regional MalwareFL and Distillation Conclusion
The Dierences
Global Features: Size 0 100 200 300 400 500 600 700
Features (#)
0
1000000
2000000
3000000
4000000
5000000
6000000
7000000
8000000
9000000
Nodes (#)
Model Size vs. Number of Tree Nodes (Global)
N=3
N=500
N=1000
N=2000
Figure: Model size for the combined dataset.Number of nodes for an increased number of
ensemble trees of increasing feature set sizes.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 16 / 52

17. Regional MalwareFL and Distillation Conclusion
The Dierences
Overview: Accuracy 0 30 60 90 120 150 180 210 240 270
Features (#)
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Accuracy (%)
Classification Accuracy vs. Number of Features (US)
N=2
N=2000
Figure: Accuracy rates for the US dataset.0 30 60 90 120150180210240270300330
Features (#)
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Accuracy (%)
Classification Accuracy vs. Number of Features (BR)
N=3
N=2000 Figure: Accuracy rates for the BR dataset.01002003004005006007008009001000110012001300
Features (#)
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Accuracy (%)
Classification Accuracy vs. Number of Features (JP)
N=2
N=2000 Figure: Accuracy rates for the JP dataset.0 100 200 300 400 500 600 700
Features (#)
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Accuracy (%)
Classification Accuracy vs. Number of Features (Global)
N=3
N=2000 Figure: Accuracy for the global dataset.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 17 / 52

18. Regional MalwareFL and Distillation Conclusion
The Dierences
Replicated Architecture
Figure: Single Model Distillation. Figure: Multiple Regional Model
Distillation.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 18 / 52

19. Regional MalwareFL and Distillation Conclusion
New Architecture
Agenda
1
Regional Malware
The Dierences
2
FL and Distillation
New Architecture
Case Study
3
Conclusion
Generalization
Final Remarks
Cross-Regional Malware Detection via Model Distilling and Federated Learning 19 / 52

20. Regional MalwareFL and Distillation Conclusion
New Architecture
Does a global model help?
Cross-Regional Malware Detection via Model Distilling and Federated Learning 20 / 52

21. Regional MalwareFL and Distillation Conclusion
New Architecture
US predicting the world 5015025035045055065075085095010501150125013501450
Features (#)
50
55
60
65
70
75
80
85
90
95
100
Accuracy (%)
Classification Accuracy vs. Number of Features (US vs. BR and JP)
BR (N=3)
BR (N=2000)
JP (N=3)
JP (N=2000)
Figure: Cross-dataset accuracy rate.Trained US model classifying the samples from the
other datasets.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 21 / 52

22. Regional MalwareFL and Distillation Conclusion
New Architecture
BR predicting the world 5015025035045055065075085095010501150125013501450
Features (#)
30
35
40
45
50
55
60
65
70
Accuracy (%)
Classification Accuracy vs. Number of Features (BR vs. US and JP)
US (N=3)
US (N=2000)
JP (N=3)
JP (N=2000)
Figure: Cross-dataset accuracy rate.Trained BR model classifying the samples from the
other datasets.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 22 / 52

23. Regional MalwareFL and Distillation Conclusion
New Architecture
JP predicting the world 5015025035045055065075085095010501150125013501450
Features (#)
50
55
60
65
70
75
80
85
90
95
100
Accuracy (%)
Classification Accuracy vs. Number of Features (JP vs. BR and US)
BR (N=3)
BR (N=2000)
US (N=3)
US (N=2000)
Figure: Cross-dataset accuracy rate.Trained JP model classifying the samples from the
other datasets.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 23 / 52

24. Regional MalwareFL and Distillation Conclusion
New Architecture
Three-Layer Architecture
Figure: Single Model
Distillation.
Figure: Multiple Regional
Model Distillation.
Figure: Regional Model
Distillation from Global.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 24 / 52

25. Regional MalwareFL and Distillation Conclusion
New Architecture
How to best build local-to-global models?
Cross-Regional Malware Detection via Model Distilling and Federated Learning 25 / 52

26. Regional MalwareFL and Distillation Conclusion
New Architecture
Enriching the Global Model 05101520253035404550556065707580859095
Portion (%)
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95
100
Accuracy (%)
Classification Accuracy vs. Dataset Portion (Base=US,BR,JP)
US
BR
JP
Figure: Building a global model.Accuracy rate for building a global model from dierent
portions of the source datasets.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 26 / 52

27. Regional MalwareFL and Distillation Conclusion
New Architecture
Enriching the US Model: Random Sampling 05101520253035404550556065707580859095
Portion (%)
75
80
85
90
95
100
Accuracy (%)
Classification Accuracy vs. Dataset Portion (Base=US)
US
BR
JP
Figure: Extending the existing US model.Accuracy rates on the dierent datasets for
dierent portions of the source datasets using random sample selection.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 27 / 52

28. Regional MalwareFL and Distillation Conclusion
New Architecture
Enriching the BR model> Random Sampling 05101520253035404550556065707580859095
Portion (%)
30
35
40
45
50
55
60
65
70
75
80
85
90
95
100
Accuracy (%)
Classification Accuracy vs. Dataset Portion (Base=BR)
US
BR
JP
Figure: Extending the existing BR model.Accuracy rates on the dierent datasets for
dierent portions of the source datasets using condence-based sample selection.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 28 / 52

29. Regional MalwareFL and Distillation Conclusion
New Architecture
Enriching the JP model: Random Sampling 05101520253035404550556065707580859095
Portion (%)
75
80
85
90
95
100
Accuracy (%)
Classification Accuracy vs. Dataset Portion (Base=JP)
US
BR
JP
Figure: Extending the existing JP model.Accuracy rates on the dierent datasets for
dierent portions of the source datasets using random sample selection.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 29 / 52

30. Regional MalwareFL and Distillation Conclusion
New Architecture
Enriching the US model: Condence-based Sampling 05101520253035404550556065707580859095
Portion (%)
75
80
85
90
95
100
Accuracy (%)
Classification Accuracy vs. Dataset Portion (Base=US)
US
BR
JP
Figure: Extending the existing US model.Accuracy rates on the dierent datasets for
dierent portions of the source datasets using condence-based sample selection.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 30 / 52

31. Regional MalwareFL and Distillation Conclusion
New Architecture
Enriching the BR model: Condence-based Sampling 05101520253035404550556065707580859095
Portion (%)
30
35
40
45
50
55
60
65
70
75
80
85
90
95
100
Accuracy (%)
Classification Accuracy vs. Dataset Portion (Base=BR)
US
BR
JP
Figure: Extending the existing BR model.Accuracy rates on the dierent datasets for
dierent portions of the source datasets using condence-based sample selection.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 31 / 52

32. Regional MalwareFL and Distillation Conclusion
New Architecture
Enriching the JP model: Condence-based Sampling 05101520253035404550556065707580859095
Portion (%)
75
80
85
90
95
100
Accuracy (%)
Classification Accuracy vs. Dataset Portion (Base=JP)
US
BR
JP
Figure: Extending the existing JP model.Accuracy rates on the dierent datasets for
dierent portions of the source datasets using condence-based sample selection.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 32 / 52

33. Regional MalwareFL and Distillation Conclusion
New Architecture
Are real models trained from scratch?
Cross-Regional Malware Detection via Model Distilling and Federated Learning 33 / 52

34. Regional MalwareFL and Distillation Conclusion
New Architecture
Self-Distillation 0 100 200 300 400 500 600 700 800 900
Features (#)
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Accuracy (%)
Classification Accuracy vs. Locally-Distilled Models (Teacher-Student)
US
BR
JP
Figure: Self-Model Distilling.Number of features required to achieve the maximum accuracy
rate for the dierent datasets.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 34 / 52

35. Regional MalwareFL and Distillation Conclusion
New Architecture
Global-to-Local Distillation 0100200300400500600700800900100011001200130014001500
Features (#)
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Accuracy (%)
Classification Accuracy vs. Globally-Distilled Models (Teacher-Student)
US
BR
JP
Figure: Global to Local Model Distilling.Number of features required to achieve the
maximum accuracy rate for the dierent datasets.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 35 / 52

36. Regional MalwareFL and Distillation Conclusion
New Architecture
Heterogeneous Distillation Features = 1100 Features = 1500 Features = 1700
Figure: RF's ensemble of dierent features set sizes.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 36 / 52

37. Regional MalwareFL and Distillation Conclusion
New Architecture
Heterogeneous Distillation 60 65 70 75 80 85 90 95 100
Dataset Portion (%)
1000
1100
1200
1300
1400
1500
1600
1700
1800
Features (#)
Ideal Feature Set Size vs. Dataset Portion (Teacher-Student)
Global/US
Global/BR
Global/JP
US/US
US/BR
US/JP
BR/US
BR/BR
BR/JP
JP/US
JP/BR
JP/JP
Figure: Global to Local Model Distilling.Variation on the number of features required to
achieve the maximum accuracy rate for dierent portions of the source datasets.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 37 / 52

38. Regional MalwareFL and Distillation Conclusion
New Architecture
What is the real impact of ML on AVs?
Cross-Regional Malware Detection via Model Distilling and Federated Learning 38 / 52

39. Regional MalwareFL and Distillation Conclusion
New Architecture
ML-derived YARA Rules
1import " pe "
2
3rule rule_from_ml_0 {
4condition :
5 pe . imports (/(.) . dll /i , / closehandle /i)
6 and
7 pe . characteristics & pe . EXECUTABLE_IMAGE
8 and
9 pe . exports (/ dllunregisterserver /i)
10}
Code 1:
Cross-Regional Malware Detection via Model Distilling and Federated Learning 39 / 52

40. Regional MalwareFL and Distillation Conclusion
New Architecture
Matching Time
Table: Matching performance.Wall time (s) for matching Yara rules derived from ML
models of dierent feature sets sizes against a real, infected lesystem.
Features 1100 1200 1300
Time 13m57s 14m00s (+0.3%) 14m05s (+1%)
Features 1400 1500 1600 1700
Time 14m50s (+6%) 15m57s (+14%) 17m58 (+29%) 19m33s (+40%)
Cross-Regional Malware Detection via Model Distilling and Federated Learning 40 / 52

41. Regional MalwareFL and Distillation Conclusion
New Architecture
Explaining Rule's Performance 1100 1200 1300 1400 1500 1600 1700
Features (#)
500
600
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
1800
Rules (#)
Number of Rules vs. Number of Features (Global-Distilled)
Malware Goodware Total
Figure: Number of rules vs. feature size.
The number of generated rules moderately
increases with the number of features.1100 1200 1300 1400 1500 1600 1700
Features (#)
23
24
25
26
27
28
29
30
31
32
33
Rules Depth (#)
Rules Depth vs. Number of Features (Global-Distilled)
Malware
Figure: Rules depth vs. feature size.The
average depth of the rules increases with the
number of features.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 41 / 52

42. Regional MalwareFL and Distillation Conclusion
Case Study
Agenda
1
Regional Malware
The Dierences
2
FL and Distillation
New Architecture
Case Study
3
Conclusion
Generalization
Final Remarks
Cross-Regional Malware Detection via Model Distilling and Federated Learning 42 / 52

43. Regional MalwareFL and Distillation Conclusion
Case Study
The Original Scenario 1 2 3 4 5 6
Time (Month)
70
75
80
85
90
95
100
Detection Rate (%)
Detection Rate over Time (No Update)
US BR JP
Figure: Detection rate as a time-series for the individual static models.Previously
trained classiers attempt to detect new threats. Performance degradation due to concept
drift is observed.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 43 / 52

44. Regional MalwareFL and Distillation Conclusion
Case Study
Drift Detection Scenario 1 2 3 4 5 6
Time (Month)
70
75
80
85
90
95
100
Detection Rate (%)
Detection Rate over Time (Drift Detection)
US BR JP
Figure: Detection rate as a time-series for the individual, drift-aware models.The
retraining of models when concept drift is detected takes the detection rate back to its original
level.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 44 / 52

45. Regional MalwareFL and Distillation Conclusion
Case Study
Drift Detection + Federated Learning Scenario 1 2 3 4 5 6
Time (Month)
70
75
80
85
90
95
100
Detection Rate (%)
Detection Rate over Time (FL Retraining)
US BR JP
Figure: Detection rate as a time-series for the globally-distilled models.The use of data
from a global model not only mitigated the drift eects but also increased the detection rate
for all datasets.
Cross-Regional Malware Detection via Model Distilling and Federated Learning 45 / 52

46. Regional MalwareFL and Distillation Conclusion
Case Study
Overview 1 2 3 4 5 6
Time (Month)
70
75
80
85
90
95
100
Detection Rate (%)
Detection Rate over Time (No Update)
US BR JP
Figure: Original1 2 3 4 5 6
Time (Month)
70
75
80
85
90
95
100
Detection Rate (%)
Detection Rate over Time (Drift Detection)
US BR JP Figure: Drift Detection1 2 3 4 5 6
Time (Month)
70
75
80
85
90
95
100
Detection Rate (%)
Detection Rate over Time (FL Retraining)
US BR JP Figure: Drift Detection + FL
Cross-Regional Malware Detection via Model Distilling and Federated Learning 46 / 52

47. Regional MalwareFL and Distillation Conclusion
Generalization
Agenda
1
Regional Malware
The Dierences
2
FL and Distillation
New Architecture
Case Study
3
Conclusion
Generalization
Final Remarks
Cross-Regional Malware Detection via Model Distilling and Federated Learning 47 / 52

48. Regional MalwareFL and Distillation Conclusion
Generalization
Extending to other feature selectors
Table: Feature Selection Method.Ideal feature set size for the multiple regional malware
datasets.
US BR JP
F-Score 290 340 800
Chi2 292 342 803
Mutual Info294 345 812
Cross-Regional Malware Detection via Model Distilling and Federated Learning 48 / 52

49. Regional MalwareFL and Distillation Conclusion
Generalization
Extending to other classiers
Table: Classier Inuenceon the detection of dierent regional malware datasets. Feature
set sizes.
95% 99%
US BR JP US BR JP
RF 35 40 45 290 340 800
SGD 35 40 45 292 342 805
AdaBoost35 40 45 292 342 805
SVM 36 41 46 295 345 813
Cross-Regional Malware Detection via Model Distilling and Federated Learning 49 / 52

50. Regional MalwareFL and Distillation Conclusion
Generalization
Extending to other distillation techniques
Table: Distillation Technique Inuenceon the detection of dierent regional malware
datasets. Feature set sizes.
US BR JP
TS 300 (+3%) 400 (+17%) 900 (+12.5%)
FMF299 (+3%) 402 (+18%) 902 (+12.5%)
Cross-Regional Malware Detection via Model Distilling and Federated Learning 50 / 52

51. Regional MalwareFL and Distillation Conclusion
Final Remarks
Agenda
1
Regional Malware
The Dierences
2
FL and Distillation
New Architecture
Case Study
3
Conclusion
Generalization
Final Remarks
Cross-Regional Malware Detection via Model Distilling and Federated Learning 51 / 52

52. Regional MalwareFL and Distillation Conclusion
Final Remarks
Thanks!
Questions? Comments?
[email protected]
@MarcusBotacin
Cross-Regional Malware Detection via Model Distilling and Federated Learning 52 / 52