Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online

When connected to the internet, it is probably just a matter of time before one of your VMs is under a DDoS attack. In this session, Wido covered how CloudStack handles this and what can you do about it.

--

The CloudStack European User Group 2024 took place on September 19th in Frankfurt, Germany. This year's event was another successful User Group for the community. The event drew a strong audience, highlighting the growth of the CloudStack community in the region.

1. Internet facing VMs
and the DDoS
problem
19.09.2024
Public

2. •Wido den Hollander (1986)
•Born and live in the Netherlands
•Married, two sons (2020 and 2022)
•CTO @ Your.Online
•Started my own hosting company in 2003
•My company became part of Your.Online in 2018
•I’m a tech guy
•Linux
•Networking
•Development
•Architecture
•Open Source & Tech
•Apache CloudStack developer and PMC member
•Cephevangelist
•IPv6 fanatic
1
Who am I?
Wido den Hollander
1) Don’t ask me, I’ll talk way too long! ;-)
2-PUBLIC -

3. Who is Your.Online?
Introduction to Your.Online
3
Your.Onlineisateamofpioneersfromalloverthe
worldunitedbythepassionofhelpingbusinesses
succeedonline.Ourteamsoflocalexpertsprovide
highlystandardizedmanagedservicestohigh-intent
customerstoreachtheirfullonlinepotential.We
cherishoursuccessfultrackrecordinacquiring,
developing,andempoweringstronglocalbrandsto
leadtheirmarkets
4 8FT E
1
3 10 FT E
5
30 FTE
2
2 00 FT E
5
60FTE
Expand intoMarket Position
5
50FTE
1
200 FTE
120 FTE
1
0

4. •We run two large Apache CloudStack deployments
•Yourhostingin the Netherlands
•Axarnetin Spain
•More deployments coming!
We love CloudStack!
Apache CloudStack @ Your.Online
Infrastructure at Your.Online
4-PUBLIC -

5. •Webservers
•DNS servers
•E-Mail
•VPS
•These are DDoS magnets!
These services all have public IPv4 and IPv6 addresses
Our services are public
Internet facing
5-PUBLIC -

6. •Layer 3: Bandwidth saturation
•Just sent 10s or 100s of Gigabits of data towards an IP
•Layer 4: TCP SYN-flood attacks
•Sent a massive amount of SYN TCP/IP packets towards an IP
•Layer7:
•DNS amplificationattacks
•DNS random record attack
•HTTP Slowloris
•HTTP flooding
•HTTP TLS attacks
•Many, many, many more
What kind of DDoS attacks?
DDoS attacks
6-PUBLIC -

7. •Firewalls protectme againsta DDoS! Nope
•They can get saturated quickly
•Bandwidth exhaustion
•Concurrent sessions
•Packets/s
We have firewalls!
Security
7-PUBLIC -

8. •Thereis alwayssomebodywitha biggergun
•Localfirewalls aren’tsufficient
The person withthisbiggergun canbea DDoS-er withjusta
bunchof machines andthehping3tool.
(Anda networkwhichdoesn’tdo outboundtraffic filtering
byimplementingBCP38)
I will buy a bigger firewall!
Security
8-PUBLIC -

9. “BCP38 is RFC2827: Network IngressFiltering: DefeatingDenialof Service Attacks whichemployIP Source AddressSpoofing. “
Best Current Practice 38: Prevent spoofed IP-packets from leaving yournetwork
IP-Address spoofing
9-PUBLIC -
•Manynetworksdo notcheck theiroutboundpacketsandallowIP-
spoofing
•CloudStack’ssecurity groupsdo preventsource addressspoofing
•ip(6)tablesusesanipsettocheck thesource of eachpacket
•Thisdoesn’thelp youwhenyouare thevictimof theattack
•Helpstopreventthatyournetworktakes part in a DDoS attack
usingspoofedpackets

10. •Security Groupsare ip(6)tablesruleson thehypervisor
•Shared Networks withKVM
•Theycontrol whichtraffic is allowedfora VM
•Ingress
•Egress
•Theyusestatefulconnectiontracking
CloudStack’ssecurity groups
Security Groups
10-PUBLIC -

11. StatelessFiltering
Eachpacketis processedin isolationwithout regardtoany
otherpackets. Traditional firewalls onlyconsiderpacket
headers (source/destinationIP, port, etc.) forfiltering.
Stateful vs. Stateless Filtering
Connection tracking
11-PUBLIC -
Stateful Filtering
The firewall keeps track of the state of network
connections. It monitors the entire session (e.g., a TCP
handshake) and applies rules based on the state of the
connection.

12. •State TableLimits: The state tablethattracks connectionshas finitelimits. On veryhigh-traffic systems (like web servers or
firewalls in large enterpriseenvironments), thistablecanbecomefull, leadingtolegitimateconnectionsbeingdropped.
•State TableExhaustion(DoSAttack): Attackerscanexploitstatefulconnectiontracking byinitiatinga large numberof
connectionstofillup thestate table, leadingtoa Denialof Service (DoS). Oncethetableis full, thefirewall maystart dropping
legitimateconnections.
What is the downside of Stateful connection tracking?
Stateful connection tracking
12-PUBLIC -

13. hping3
Stateful connection tracking
13-PUBLIC -

14. hping3
Stateful connection tracking
14-PUBLIC -

15. State TableExhaustion
Stateful connection tracking
15-PUBLIC -

16. •ThisaffectsallVMsrunning on thesamehypervisor
•Onlyifyouare usingShared Networks
•The HV thenperformsthefiltering locallyusingip(6)tables
•In VPC networkstheSG runs on theVR
•Thissuffers even more fromthisproblem!
•I have limitedexperiencewithVPCs
•It canaffect connectionsfromandtothehypervisor
•NFS, Ceph& iSCSI
•Youcan’tjustkeep increasingthelimit
•Withhundredsor thousandsof connectionsper second youwillrun out of memory veryquickly
State TableExhaustion
Stateful connection tracking
16-PUBLIC -

17. •Don’thost DDoS ‘magnet’ services
•ToRexit nodes, Minecraftservers, UnmanagedVPS, Certainwebsites, etc
•Investin upstream Anti-DDoS services
•Cogent, Arelion, Colt andothershave these services
•ImprovetheSecurity Groupsin CloudStack
•Tell usmore!
Whatcanwe do aboutthis?
Solutions
17-PUBLIC -

18. •We can’tfix everythingwithCloudStack
•SomeDDoS attacks are justtoobig andwilloverwhelmyournetwork
•Manyattacks canhoweverbehandledbyyournetwork
•But it’syourLinux hypervisor whichcan’tkeep up withthepackets/second
•We couldmake theSecurity Groupsmodularin theKVM Agent
•The default driver wouldusesecurity_group.pyas itdoes now
•Additionaldrivers couldusedifferent methodstoachievethesameend result
ImprovingCloudStack
Solutions
18-PUBLIC -

19. Driver options and capabilities
Security Groups
eBPF Hardware Packet Filtering
•First line if defense is stateless
•We carefully select which packets to
inspect stateful
•And thus track the connection
Tiered Filtering
•Certain NICs support this
•Mellanox
•Intel
•Requires implementation per
vendor/situation?
•Extended Berkeley Packet Filter
•Build-in in Linux kernel since 3.18
•Run custom code in the kernel which
can do packet filtering
•High-performance packet filtering
•Integration with iptables is available
•bpfilteris on it’s way
1
•Easy to plug your own solution on the
hypervisor-level
•Implement a driver which suits your
specificsituation/needs
Custom Solution
19–PUBLIC –
A driver might need to use a combination of the options mentioned here
1)-https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/

20. •I don’tthinkso
•Thiswillneedproper investigation
•Currentimplementationcanbeimproved
•Modularitywouldhoweverbeverywelcome
•Anyvolunteers?
Is is thateasy?
Solutions
20-PUBLIC -

21. •I won’tmake attackerssmarterthanneeded
•We usea combinationof things
1.Flow analysis on incomingpackets
•Juniperrouters providesFlowdata
•FastnetMonanalyzestheflows
•We canfilter a specific/32 (IPv4) or /128 (IPv6)
2.Hardware firewall machines optimizedtofilter specific, common, DDoS attacks
•A single destinationIP-addresscanbefilteredbythese machines
•These machines are strategicallyplacedin ournetworkandhave high capacityconnections
3.In case it’stoomuchwe offloadtoexternalcarriers
•Theythando theirmagic
Whatdoes Your.Online do?
Solutions
21-PUBLIC -

22. •[email protected]
•@widodh
•https://blog.widodh.nl/
Thanksforlistening!
Thank you!
22-PUBLIC -