Breaking the Kubernetes Kill Chain: Host Path Mount

Microsoft's Threat Matrix for Kubernetes helps organizations understand the attack surface a Kubernetes deployment introduces to their environments. This ensures that adequate detections and mitigations are in place. By covering over 40 different attacker techniques, defenders can learn about Kubernetes-specific mitigations and controls to deploy to their environments. In this session, we will explore the MS-TA9013 Host Path Mount technique, which is commonly used by attackers to perform privilege escalation in a Kubernetes cluster. Attendees will learn how attackers and defenders can:

* Escape the container's host volume mount to gain persistence on an underlying node
* Move laterally from the underlying node into the customer's cloud environment
* Analyze Kubernetes audit logs to detect pods deployed with a hostPath mount
* Deploy an admission controller that prevents new pods from using a hostPath mount

1. SANS Oslo –Tuesday 23rdApril 2024
Eric Johnson & Ryan Nicholson
Breaking the
Kubernetes Kill Chain:
Host Path Mount

2. SANS Oslo April 2024 Community Night
ŒIntroductions
Kubernetes Threat Matrix
ŽExploit: Host PathMount
Detection: Kubernetes Audit Logs
Prevention: Kubernetes Admission Control
‘Conclusions
Agenda

3. Eric Johnson
•Principal Security Engineer, Puma Security
→Coder: cloud infrastructure automation, CI / CD orchestration, cloud architecture, security tool automation
→Security assessments: cloud, dev/sec/ops, source code, web apps, mobile apps
•Senior Instructor, SANS Institute
→Contributing author of SEC540, SEC510, SEC549
•Community, Training, Education
→AWS Community Builder, GPCS, GSSP, GWAPT, AWS Dev, CISSP
→Iowa State M.S. Information Assurance, B.S. Computer Engineering
•Contact information
→LinkedIn: https://www.linkedin.com/in/eric-m-johnson/
→Email: [email protected]
$ awsstsget-caller-identity
3

4. Introduction
4
Ryan Nicholson
•Owner, Blue Mountain Cyber, LLC
→Cybersecurity consulting
→Cloud Security Assessments
•Senior Instructor, SANS Institute
→Author of SEC488: Cloud Security Essentials
→Co-Author of SEC541: Cloud Security Attacker Techniques, Monitoring, and Threat Detection
•Contact information
→LinkedIn: https://www.linkedin.com/in/%F0%9F%A4%99-ryan-nicholson-7aa11563/
→Email: [email protected]

5. Kubernetes Threat Matrix

6. SANS Oslo April 2024 Community Night
Cloud Managed Kubernetes Services
The major cloud providers offer managed Kubernetes services to help customers
run workloads without the operational responsibility of managing the cluster control
plane:
Google
Kubernetes
Engine (GKE)Azure Kubernetes
Service (AKS)
AWS Elastic
Kubernetes
Service (EKS)
6

7. SANS Oslo April 2024 Community Night
Microsoft Threat Matrix for Kubernetes
Image source: https://www.microsoft.com/en-us/security/blog/wp-content/uploads/2021/03/Matrix.png

8. SANS Oslo April 2024 Community Night8
Kubernetes Cluster Components

9. SANS Oslo April 2024 Community Night
•Kubernetes worker nodes run a host
operating system (e.g., Google COS,
Amazon Linux 2, Amazon Ubuntu, RHEL)
and a container runtime
•Container runtimes, containerd, provide
isolation for containers running on the
worker node
•Kubernetes resources are managed by
the kubelet running on the host through
containerd
•Pods directly using the host's
namespace or file system can bypass
container security controls
Kubernetes Worker Node Components
•9

10. SANS Oslo April 2024 Community Night
Demo: ACE135 Aviata EKS Infrastructure

11. Exploit: Host Path Mount

12. Step 1: Retrievekubeconfig
12

13. Step 2: Submit Malicious Manifest
13

14. Step 3: Image Pull
14

15. Step 4: Rogue Pod Deployed and Communicating!
15

16. SANS Oslo April 2024 Community Night
Demo: Rogue K8s Manifest with hostPath Mount and Reverse Shell Payload

17. SANS Oslo April 2024 Community Night
Demo: Receiving Reverse Shell and Stealing Node Credentials

18. SANS Oslo April 2024 Community Night
Demo: Using Stolen Credentials

19. Detection: Kubernetes Audit Logs

20. •Initial detections could include:
oUnusual outbound traffic from EKS node (AWS VPC Flow Logs)
oNew, previously unseen image
•Which raises other questions:
oIs this traffic tied to a pod or a process on the node?
oWhat is the payload (if visible)?
oIf a rogue pod...
§How did it get there?
§When did it start?
§Is it still running?
§Who deployed it?
§What are the deployment specifics (e.g., image, volume mounts)?
Detecting the Rogue Container
20

21. SANS Oslo April 2024 Community Night
Demo: Unusual Network Traffic

22. SANS Oslo April 2024 Community Night
Demo: Host Path Mount Exploit Detection

23. Prevention: Kubernetes Admission
Control

24. SANS Oslo April 2024 Community Night
Kubernetes Admission Controllers
Image Source: https://www.armosec.io/blog/kubernetes-admission-controller/
24

25. SANS Oslo April 2024 Community Night
•Flexible policy engine for cloud-native
configuration and application authorization
•Written using the Rego language to declare
policy for Kubernetes, Docker, Terraform,
CloudFormation, Kong Gateway, Envoy, and
many more tools
•OPA policies are fed to an integration, which
evaluates and enforces each policy
•Kubernetes Gatekeeper admission controller
evaluates Open Policy Agent (OPA) rules
•Gatekeeper installs Custom Resource
Definitions (CRD) for Constraints and
ConstraintTemplates objects
•Gatekeeper Library includes several pre-
built, parameterized security policies
Open Policy Agent & Gatekeeper
25
Open Policy Agent (OPA)OPA Gatekeeper

26. SANS Oslo April 2024 Community Night
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
…
spec:
crd:
spec:
names:
kind: K8sPSPHostFilesystem
validation:
…
properties:
allowedHostPaths:
type: array
description: "An array of hostpath objects, representing paths and read/write
configuration."
…
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8spsphostfilesystem
import data.lib.exclude_update.is_update
…
volume := input_hostpath_volumes[_]
allowedPaths := get_allowed_paths(input)
input_hostpath_violation(allowedPaths, volume)
msg := sprintf("HostPath volume %v is not allowed..)
Gatekeeper Library Host File System Constraint Template
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

27. SANS Oslo April 2024 Community Night
Create a new Gatekeeper constraint
using the K8sPSPHostFilesystem kind:
•Set the enforcementAction to deny to
prevent resource creation
→Use dryrun and warn for testing
constraints and passively capturing
alerts
•Set the namespaces array to scope
the constraint to specified
namespaces (optional)
•Set the allowedHostPaths parameter
to an allow list of mount prefixes and
r/w capabilities
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPHostFilesystem
metadata:
name: aviata-host-mounts
spec:
enforcementAction: deny # [dryrun | warn | deny]
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces:
- "default"
parameters:
allowedHostPaths:
- pathPrefix: "/mnt/data/aviata"
readOnly: false
Gatekeeper Host File System Constraint
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

28. SANS Oslo April 2024 Community Night
Demo: Gatekeeper Host Path Mount Admission Controller

29. Conclusions

30. SANS Oslo April 2024 Community Night
Closing Remarks
As cloud migrations continue, it is becoming increasingly important
for security professionals to understand cloud-native security:
30
Learn how operate Kubernetes
and container services.
Attack cloud-native workloads
using the MITRE ATT&CK and
Kubernetes threat matrices.
Detect compromised workloads
using cloud-native and public cloud
log sources.
Prevent cloud-native and
application misconfigurations
using policy as code.

31. SANS Oslo April 2024 Community Night
https://sans.org/ace135

32. SANS Oslo April 2024 Community Night
CHAPTER 1 FLIGHT PLAN
•List Agenda Sections
→Second level
–Third Level