Using Metrics for Fun, Developing with the KV Store + Javascript & News from Conf 2018! (Security, ITOps & More!)

We explore "Metrics, mstats and Me: Splunking Human Data” and also have some insights into the KV Store and javascript use in dashboards. We’ll also re-cover the conf18 updates for those who couldn’t attend our last session.

1. Splunk User Group Edinburgh

2. Harry McLaren Managing Consultant at ECS Security Splunk Enablement Lead & Member of Splunk Trust Leader of the Splunk User Group Edinburgh

3. Introduction to ECS Security Splunk Partner - UK Security Consultancy & Managed SOC Provider Splunk Revolution Award & Splunk Partner of the Year

4. Agenda Housekeeping: Event Overview & House Rules Metrics, mstats and Me (Andrew McManus) KV Store & Javascript (Mark Hunter) Splunk .conf18 Updates (Harry McLaren) Security IT Ops Others (Docker)

5. Splunk [Official] User Group “The overall goal is to create an authentic, ongoing user group experience for our users, where they contribute and get involved” Technical Discussions Sharing Environment Build Trust No Sales!

6. Human Data to Splunk Metrics, | mstats And Me Edinburgh Splunk User Group – 22/11/2018 Andrew McManus – Associate Security Consultant - ECS

7. About Myself Associate Security Consultant at ECS Prior - Senior/Security Operations Center Analyst at ECS (Non-Pearson) Credentials: Admin, Sales Rep I Know a lot about searches. Like to mess about with shiny new Splunk Additions Type 1 Diabetic – Since 2001. Part Cyborg

8. Diabetes 101? Isn’t this a Splunk talk? Body uses Insulin to regulate glucose between blood stream and cells Type 1 – Something causes destruction of insulin cells in Pancreas, causing deficiency. No-one’s sure about exact cause – widely believed to be Auto-Immune related. Manual injections required. Manual glucose testing required. Type 2 – Resistance to Insulin, normally through diet or environmental aspects. Can go into remission with treatment/diet. Can use injections or pills to regulate glucose content.

9. Disclaimer I’m not intentionally advocating treatments or products. There are pros and cons to the products/treatments mentioned Price, usability, comfort, reaction times … Yes, I can eat sugar. Common misconception. I shouldn’t, but that’s on me. I crave dessert too much. Go to your GP if you have health concerns.

10. Measurement sTePS (until 23/07/2018) Glucose sample from blood, via a finger pricker . Glucose meter takes static snapshot of blood glucose concentration Sample taken before major meals, and ad-hoc if required Insulin taken as a response to glucose result, or recommended dosage Aiming for between 4mmol/l and 10mmol/l glucose concentration.

11. Measurement Steps (since 23/07/2018) Prescribed Abbott Libre FreeStyle sensor (other sensors available) Checks glucose content in interstitial fluid below skin, not blood Takes reading every 1m and calculates trending behaviour . Retains a rolling 8 hours worth of data on sensor Transfers readings to monitoring device, or phone, via NFC

12. A quick demonstration Hope the Pizza and Beer don’t shame my glucose levels …

13. Less Medical, more metrical Please! Sensor is continuously taking metric data of glucose concentration in body. Phone or meter can send this metrics data to a cloud service for doctors to see Cloud service provides a export of metric data to local machine. Metric data is Machine Data Splunk likes Machine Data – Splunk has special metrics gizmos baked in.

14. Splunk Metrics Meant for “collecting, investigating, monitoring, and sharing metrics from your technology infrastructure, security systems, and business applications in real time” Fast statistical results and visualizations using command Splunk commands Can’t search for events, in traditional sense (i.e. security logs). Claims: 20x faster than equivalent accelerated log ( tstats ) and 200x faster than non-accelerated logs/event data searches. What makes up a Metric?

15. Splunk Metrics Timestamp Timestamp of metric Metric Name dotted namespace i.e. server.www1.response.5xx Value Numerical data point Dimensions Metadata to describe data – i.e. AWS AZ, server name, technology name Can have multiple dimensions

16. Getting Metrics in Various methods. | mcollect , HEC, statsD , collectD , csv to metric, Insights for Infra App

17. CollectD https://collectd.org Periodically collects system and application performance metrics. Point collectd’s write_http module to HEC with collectd_http sourcetype Quick Demo – Computer Metrics

18. Metrics from LAptop Hint: Don’t be like me. Use Splunk App for Infrastructure ( https://splunkbase.splunk.com/app/3975/) Sets Collectd up for you.

19. Diabetic Data to Metric_CSV Data needed to be transformed to match metric_csv sourcetype Quick dirty Python Script to import csv, transform timestamps and collapse data to expected fields Write to new file and ingest this on a monitor input Danger – No “| delete” method for metrics – once it’s in, it’s in. Keep in mind if monitoring a file, or one- shotting data in.

20. Quick Code Review

21. Mcatalog List metric names, hosts and dimensions Useful to see what metrics you have in Splunk | mcatalog values(_dims) values(host) by metric_name

22. MStats Run statistical commands on metric values. | mstats avg (_value) as ” avg_glucose " WHERE metric_name =" personal.glucose.historic " AND "index"=" diabetic_data " span=1h | append [| mstats sum(_value) as total_quick_insulin WHERE metric_name =" personal.insulin.rapid.dose.units " AND index= diabetic_data span=1h] | append [| mstats sum(_value) as total_carbs WHERE metric_name =" personal.carbohydrate.grams " AND index= diabetic_data span=1h]

23. Viewing Metrics – Metrics Explorer App on Splunkbase – will be added to core Splunk eventually.

24. New! from CONF 2018: Metrics Workspace Download from SplunkBase : One stop shop for metric discovery, dash-boarding and alerting. No SPL required

25. Further Reading .conf2018: Getting logs and metrics into metricstore ( https://conf.splunk.com/files/2018/recordings/getting-logs-and-metrics-fn1888.mp4) New Splunk Metrics Workspace Experience ( https://conf.splunk.com/files/2018/recordings/exciting-to-be-announced-fn1508.mp4) .conf2017: Getting Metrics In: Splunking Metrics – The Right Way ( https://conf.splunk.com/files/2017/slides/getting-metrics-data-in.pdf)

26. Any Questions?

27. THE KVSTORE FOR FUN AND PROFIT * * Profit not guaranteed ** ** Fun not guaranteed either

28. ELEGANT CAT, SITTING This is my cat. His name is Roran . He also answers to “Catface.” I call him this because his face bears a quite uncanny resemblance to the bewhiskered visage of a cat. Also ”Roy Cattersley ”, “ Catweazel ” and “The Floofmeister .” When my waffle becomes intolerable, think back to his fluffy coat, his furry paws, his gentle smile. It’ll all be over in no time. One way or another.

29. PART 1: CSV vs KV STORE. FIGHT!

30. EXTENDED CHAT summarised As the title hints, I’m going to talk about the KV Store : How KV Store collections differ from CSV collections How to quickly deploy KV store collections How to take advantages of what they offer Quick look at an in-development SimpleXML -extended KV Store dashboard

31. EXPLANATORY CSV SLIDE CSV lookup queries look like this: index=foo sourcetype =alignment | lookup detectEvilLookup isEvil | where isEvil =1 | inputlookup detectEvilLookup where characterClass =“Fighter” | inputlookup detectEvilLookup where characterClass =“Fighter” | outputlookup theworstLookup

32. EXPLANTORY C..ER..KV-STORE SLIDE On the other hand, KV Store lookups look like this: index=foo sourcetype =alignment | lookup detectEvilLookup isEvil | where isEvil =1 | inputlookup detectEvilLookup where characterClass =“Fighter” | inputlookup detectEvilLookup where characterClass =“Fighter” | outputlookup theworstLookup append=f

33. EXPLANATION COMING, STAT No real difference in addressing them. CSV files reside on indexers, KV Store on search heads. CSV files can only append to or replace file ; KV Store can add, upsert , and delete specific field entries. KV Store has REST endpoint access. KV Store can enforce data types.

34. ENOUGH! COMPARE SYSTEMS Collections Records Fields _key Tables Rows Columns Primary Key

35. EXAMPLE CASES & SITUATIONS Better performance with a larger or frequently updated record set Any record management system – inventory, control lists, etc Preserving application state Scratchdisk (Field acceleration!) Porting

36. PART 2: SET-UP AND IMPLEMENTATION

37. ELUCIDATE CLEAR STEPS System set-up tasks Configuring a collection Dashboards and logic

38. EGADS! CAT SHENANIGANS. I thought you might like to be reminded of Catface.

39. http://downloads.jordan2000.com/ splunk /

40. EASILY CONFIGURED – SEE! What do you need? Two conf files: collections.conf and transforms.conf in a search head app You can do this in the GUI, but we are not teh n00blets el oh el

41. EXAMPLE CONF SETUP (1)

42. EXAMPLE CONF SETUP (2)

43. EGREGIOUSLY CATASTROPHIC SUGGESTION Let’s live dangerously and try jumping straight to an example.

44. Part 3: FROM THEORY TO (BADLY IN NEED OF) PRACTICE

45. EXPECTED CONVERSATIONAL SLOG The problem The config The dashboard The javascript

46. Explanatory contextualising statement Replace an existing inventory and control management system Based on copied and pasted excel sheets Frequently updated daily Potentially large updates Referenced by many apps for gatekeeping

47. EXTENDED COMMENTARY SECTION 1 Collection LIVE SHOWING. YOLO!

48. EXTENDED COMMENTARY SECTION 2 Dashboard LIVE SHOWING! LIVE DANGEROUSLY!

49. EXTENDED COMMENTARY SECTION 3 Javascript LIVE SHOWING! I’ve run the JOKE into the GROUND! Gains over CSV – any?

50. EVENTUALLY (COMING SOON) Custom renderer Monitoring and troubleshooting tools Current client view of MC: ”supping from the very bladder of Satan.” A direct quote.* * Not a direct quote

51. EmBARKING CAREFULLY=SUCCESS Some things to be aware of: Export and import is everything or nothing Use CSV to export and import individual collection. IRONY. Auto lookups Switch replicate to true in collections.conf stanza You’re on the indexers now though Filtering with where Declare _key

52. THANKS, All. THALL. Feedback pls

53. Splunk .conf18 Updates Harry McLaren

54. Introducing Splunk Enterprise Security 5.2 Generally Available: 16/10/18

55. Event Sequencing Define Attacker Techniques via Multiple Matching Events The Event Sequencing Engine runs as a real-time search and listens for incoming notable events and risk modifiers that are triggered by correlation searches. Transitions can also be configured to aggregate notable events or risk modifiers that may happen after a transition match is found.

56. Event Sequencing Define Attacker Techniques via Multiple Matching Events

57. Use Case Library ES Content Updates Type Function Integrated

58. Investigation Workbench Two New Artifact Types - File Name & URL

59. Introducing Splunk Phantom Version 4.0 Security Orchestration, Automation, & Response (SOAR) Platform Clustering support for added performance and redundancy Enables Phantom to scale horizontally using additional instances for added performance and redundancy Indicator View for threat intelligence style analysis Provides a new and important way to visualize security data on the Phantom platform. Data is presented in the view organized by indicator, versus event, for easier threat-intelligence style analysis. Native Splunk search support Splunk is now the default search engine shipped with the Phantom product. Users are able to use their existing or new external Splunk instances to achieve a single source for security data storage. Elasticsearch engine remains an external option for those that prefer it.

60. Introducing Splunk User Behaviour Analytics 4.2 Generally Available: 16/10/18 User Feedback   for machine learning models  provides anomaly customization and improved threat detection accuracy Improved data ingestion   performance by up to 10x , with the new Splunk-to-Kafka UBA ingestion connector. Kafka ingestion does not require UBA to run real-time indexed search queries on core Splunk, rather uses micro-batched queries. Native single-sign-on authentication support  for multiple identity providers Okta , Microsoft ADFS and Ping Identity

61. Introducing Splunk ITSI 4.0 Predictive Analytics for Real-Time Insights KPI Predictions  We’re excited to deliver deeper insights into a potential health degradation with KPI Predictions. These utilize the breadth of data in the platform to help predict KPIs like customer experience, application workload, and infrastructure health, in order to identify issues or outages in advance. Predictive Cause Analysis  This new feature helps you drill down into the specific services underlying a predicted issue to proactively remediate and resolve it before customer experience is impacted.

62. Introducing Splunk SmartStore Cut the Cord by Decoupling Compute and Storage Allowing compute and storage tiers to be independently scaled. Automatically evaluates users’ data access patterns to determine which data needs to be accessible for real-time analytics and which data should reside in lower cost, long-term storage.

63. Introducing Dynamic Data: Active Archive Data Retention Options in Splunk Cloud Data Management Splunk provides complete lifecycle management of the archive on your behalf and remains the custodian of your data. Just like your Active Searchable data, Splunk manages all aspects of archive availability, durability, security and privacy requirements on your behalf. Data Restore Enables you to request a slice of your data to be restored back into your Splunk Cloud instance. The entire workflow is fully integrated into Splunk Web so your archived data is available at your fingertips with predictable time between retrieval to search.

64. Other Features! Selection of Interesting New Releases! Dark Mode  heightens visual contrast within Splunk dashboards. Workload Management  enables users to prioritize the allocation of compute and memory resources used by Splunk on searches and alerts to ensure users’ most critical analytics are completed first. Guided Data Onboarding  is a new graphical user interface helping customers move data into Splunk Cloud or Splunk Enterprise and guiding them through the best onboarding methodology based on their specific architecture. Logs to Metrics  helps configure and convert log events to metrics, enabling users to take advantage of breakthrough performance when monitoring and alerting on metrics with the Splunk platform. Health Report  gives Splunk administrators immediate visibility into the overall health status of their Splunk environments.

65. Introducing Splunk Next Splunk Works the Way Your Data Works Feedback from Splunk Customers Make it easier to access data with Splunk no matter where it lives or what format it is in. Make it easier to automate the actions and outcomes in order to drive the business forward. Make it possible for all kinds of people to ask questions of Splunk and get to answers, no matter their role or where they might be in the world. What Does Splunk Next Do For You ? Ask Questions: Open customers to a broader set of data sources. Get Answers: Empower a broader set of customers from IT and Security to Lines of Business. Take Action: Operate on data wherever it lives.

66. Splunk Next Experimental, Pre-release Features (Alpha/Beta) Splunk Developer Cloud: Write Splunk applications natively in the cloud. Splunk Business Flow: Analytics-driven approach into customer/user’s interactions and identify ways to optimize those interactions and processes. Splunk Data Fabric Search: Seamlessly search across massive amounts of data and federated searches across multiple instances. Splunk Data Stream Processor: Refine, modify and adjust data mid-stream and within milliseconds before the data reaches its destination. Splunk Cloud Gateway: Secure cloud service with end-to-end encryption for easy mobile engagement through a simple to install Splunk app for Mobile. Splunk Mobile: Actionable alerts and mobile-friendly dashboards on mobile devices through our Splunk Mobile App. Splunk Natural Language: Query a system and ask question of Splunk without knowing SPL Splunk TV: View Splunk on any peripheral device instead of having to purchase a dedicated PC Splunk Augmented Reality : Enjoy direct access to the Splunk dashboard and live augmented reality Splunk-powered gauges on top of real-world objects.

67. Splunk on Docker Containers are now a First-Class Citizen Splunk Support now covers Splunk Enterprise 7.2 deployments in Docker containers, enabling customers to quickly deploy and scale Splunk based on their organizations’ demands.

68. Get Involved! Splunk User Group Edinburgh https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html https://www.linkedin.com/groups/12013212 Splunk’s Slack Group Register via http://splunk-usergroups.signup.team/ Channel: # edinburgh Present & Share at the User Group? Connect: Harry McLaren | [email protected] | @cyberharibu | harrymclaren.co.uk ECS | [email protected] | @ECS_IT | ecs.co.uk