Blue-Team-Summit-2023_Social-Engineering-your-Metrics_Joe-Gray.pdf

Social-Engineering by Joe Gray

1. Social Engineering Your
Metrics: Using Data Science to
Provide Value in Reporting
Joe Gray @C_3PJoe
Principal Trainer, The OSINTion
https://linktr.ee/theosintion
1

2. •Founder, Principal Instructor, Principal Content Developer at The OSINTion
•Served in the US Navy, Navigating Submarines
•CISSP-ISSMP, GSNA, GCIH, OSWP
•Previous Forbes Contributor
•Freelance author having written for AlienVault/AT&T Cybersecurity, Dark
Reading, ITSP, Tripwire, and CSO Online
About Me
2

3. Practical Social Engineering
•Authored Practical Social Engineering via No Starch
Press:
•Via Amazon:
https://osint.mobi/pse-amazon
•Via NoStarch Press:
https://osint.mobi/pse-nostarch
3

4. The thoughts and opinions in this presentation do not
necessarily reflect those of my employers - past,
present or future.
4

5. Objectives
•Discuss Current General Metrics
•Social Engineering-specific Metrics
•Problem with Metrics
•Introduction and Application of New Metrics
•Analysis Models
•Potential for Future Integration
5

6. Current Infosec Metrics
•Password Metrics
•Breach Metrics
•Response Metrics
•Dwell Time
•ROI
6

7. Password Metrics
•Length
•Complexity
•Age
7

8. Breach Metrics
•Number of assets
•Password reuse
•Number of breaches
•Ratio of users in breaches
8

9. Response Metrics
•Mean Time to Respond (MTTR)
•Number of Incidents
9

10. Dwell Time
•How long an adversary remains
hidden and dormant before being
detected (voluntarily or
involuntarily)
10

11. ROI
•Ratio of expenditures versus
forecasted or perceived losses
11

12. Social Engineering Specific
Metrics
•Opens
•Clicks
•Credentials Stolen
•Averages or Ratio of:
•Opens
•Clicks
•Stolen Credentials
12

13. What is the
Problem?
13

14. These Metrics Don’t Really
Measure Much
14

15. Logic
Behind
this
Statement
•Emails are meant to be
opened.
•Many training platforms fall
short on detonating links
•Most non-tech folks use the
same password or a slight
variant
15

16. What am I
getting at?
16

17. Caring about
opens:
•Encourages punishing people for opening emails
•Encourages naïve and already problematic employees may use this to avoid work
(although a slippery slope)
•Doesn’t accurately address risk
17

18. Caring about
clicks:
•Is more relevant than opens
•Tests the human, but if defense in depth is implemented, should be minor risk
•Should be a test of technical solutions in addition to a test of people
• Companies Managers often miss this point
18

19. Caring about
stolen credentials:
•Is predicated on failing to have a password policy
•Holds weight due to password reuse
•Is often mitigated by using MFA and/or password managers
•Can lead to scope creep and illegal activity when personal accounts become
targeted
19

20. TL;DR
•Organizations often:
•Put the burden entirely on non-
technical employees
•Fail to see that technology can help
mitigate but not entirely solve the
problem
•Fail to admit their own shortcomings
20

21. New Ratio Metrics
That Matter (IMO)
•Open to Click Ratio
•Open to <Action>
•Open to Reporting
•Click to <Action>
•Click to Reporting
•<Action> to Reporting
21

22. Open Metrics That
Matter (IMO)
•Open to Click Ratio
•Open to <Action>
•Open to Reporting
22

23. Click Metrics That
Matter (IMO)
•Click to <Action>
•Click to Reporting
23

24. Action Ratio Metrics
That Matter (IMO)
•<Action> to Reporting
24

25. Distance
Metrics That
Matter
(IMO)
•Open to <Action>
•Click to <Action>
•Open to Report
•Click to Report
•<Action> to Report
25

26. Distance Metrics
That Matter (IMO)
•Relevance:
•While we want to avoid an incident altogether, we also want to be notified of a potential incident as early as possible.
•We don’t want to be notified of a ransomware infection by Brian Krebs, the FBI, or the ransom demand
•Using concepts from Time Based Security (h/t Winn Schwartau), we don’t need to focus on complete eradication.
•Focus on building defenses that protect until detection and remediation can occur
•TL;DR:
•How much lead time does the team have to enact IR activities?
•SHIFT LEFT!
26

27. What if I am a
consultant?
27

28. Consulting Metrics
•You can provide even more value to your clients in the community.
•Per engagement Metrics
•Client only (unless approved for release or heavily anonymized)
•Sliced metrics (per engagement across all clients)
•Population Metrics:
•Samples per client (all engagements with a single client)
•Full Population (all clients; all engagements)
28

29. Basic Statistic
Measurements
•Mean (Average): Central tendency of a data set
•Median: Point directly in the middle of the data set
•Standard Deviation: The amount of variance between data points within the dataset
29

30. Questions?
30

31. Questions?
@C_3PJoe | @TheOSINTion
https://nostarch.com/practical-social-engineering
https://osint.mobi/discord
https://tidbit.theosintion.com
https://www.linkedin.com/in/joegrayinfosec
https://linktr.ee/theosintion
31

32. •You can use code ANTISYPHON for 15% off the remainder of live courses for 2023, before The OSINTion moves to Antisyphon:
•August 26, 1200– 2030 (ET) Intelligence Investigations: Business
•https://osint.mobi/august-business-course
•September 23, 1200 – 1600 (ET) IMINT
•https://osint.mobi/september-imint-course
•September 23, 1700 - 2100 (ET) OSINT Using Recon-ng
•https://osint.mobi/september-recon-course
•October 7, 1200 – 2030 (ET) Intelligence Investigations: People
•https://osint.mobi/october-people-course
•November 4, 1700 – 2100 (ET) AASEI
•https://osint.mobi/november-aasei-course•November 4, 1700 – 2100 (ET) AITI
•https://osint.mobi/november-aiti-course
32

33. 33