mnNOG 5: Open source SD-WAN

APNIC Senior Network Analyst/Technical Trainer Dave Phelan presents on open source SD-WAN at mnNOG, held in Ulaanbaatar, Mongolia from 18 to 23 September 2023.

1. 1
Open source SD-WAN
One man’s decent into Madness
Dave Phelan - APNIC

2. 2
Who Am I?
•Dave Phelan
–Network and Infrastructure engineer for a LONG time
–Trainer at APNIC
–Parent to 2 Human children and 2 Fur Children
–Likes Cat memes
• [Date[
• [xx]

3. 33
What are we going to talk about?
•Why am I talking about ANOTHER overlay network?
•What is SD-WAN?
•What are the “standards” for this?
•What are my FOSS options?
•How do I do it?
•Should I do it (or what problem am I solving)?

4. 4
Why am I talking about this?
•Post training surveys
–Most requested content
•Finding solutions that don’t have a vendor lock in
–This is HARD…Or is it?
•Like it or not, SD-WAN is being deployed

5. 5
What is SD-WAN
•Defined in MEF-70(07/2019) Updated MEF-
70.1(11/2021)
•SD-WAN Is a Virtual Overlay Network
•Operates over one or more underlay (Layer 3)
services
•Centralised Management and Orchestration
–Usually via a Vendor Portal
•Provisions for Flexible routing
–Application based routing (YMMV)
–Load balanced/Preferred/failover etc

6. 6
What is SD-WAN – Components - 1
•Underlay Network
–The network that our SD-WAN sits over the top of.
–Can be any form of connectivity as long as we have L3
•LTE/Ethernet/MPLS/Commodity BB etc
•Overlay Network
–Virtual Tunnels for our SD-WAN Network
•SD-WAN Edge Device
–Serve as endpoints for connectivity to the Virtual Fabric
–Encapsulate and forward the traffic based on Policies

7. 7
What is SD-WAN – Components - 2
•SD-WAN Controller
–Manages and Orchestrates the Overlay Network
–Policy/routing definition is done here
•Management and Orchestration
–UI into the controller
–Allows for configuration of our Edges and creation of policy/routing

8. 8
What is SD-WAN – Components - 3
MEF-70.1 Page 13

9. 9
What are the Standards?
•MEF-70.1
–It defines the components, features, and Framework
•https://www.mef.net/resources/mef-70-1-sd-wan-service-attributes-and-service-
framework/
–Vendor Interop is questionable(non-existent)
•You have to drink the kool-aid
–It covers off the “Do” and “Don’t”, “Must”, and “Should”
–It’s all about the Subscriber and Supplier
•Not so much about the “how”

10. 10
What are my FOSS Options?
•There are now many options
–Zero-Tier
–Headscale(Based on Tailscale)
–Flexiwan
–Zevenet
–VyOS
–Others……
•BUT They all have drawbacks
–Limited Options for self-hosted controller/UI
–Still broken interop
–Not all SD-WAN Features are implemented
–Some still require you to create an account(phone home)

11. 11
How do I do it?
•Choose an Open source Option
•Install the required software on your network devices
–This is where the problems start
•Configure your routing policy
–This is where more problems occur
•Join your network devices to your Virtual Network
•Magic Magic…
•Packets go from A to B

12. 12
How DID I do it?
•Problem 1 – Network Hardware
–Low Cost, but flexible
–SD-WAN parts already there
• or that I can Modify
•Problem 2
–Which FOSS solution to use?
–Does my SW Choice drive my hardware or Visa-Versa
–What features am I missing?
–What can I do without?
–I don’t want to have to create a login with a Vendor!

13. 13
How DID I do it?
•Hardware/Network OS
–Mikrotik ROS7
–ARM Hardware has options for Zero-Tier by additional NPK
–X86 (CHR) supports docker containers
•Custom Rolled my own Docker ZeroTier Container
–More to come on this…
•Software
–ZeroTier (https://www.zerotier.com/)
–Many Deployment options
•Clients for Windows/Mac/Android/iPhone/Linux
–Can Be run as a docker container
–Doesn’t need to connect to the Mother ship (Planet servers)

14. 14
How DID I do it?
•Other Options could be
–OpenWRT
–Teltonika
–Protectli(Running OpenWRT)
•Still investigating these Options
•Still Investigating the other Software as well

15. 15
How DID I do it?
•Caveats
–This Method breaks the ties to the ZeroTier Roots
•You CAN’T do this if you run Android/iOS clients
•You CAN do this if you are running a docker/linux/wrt image
•IF you want to use Android/iOS clients, you will need create a ZT login and
NOT REMOVE/DISABLE the planets
–Packet Processing is done in CPU
•No HW offload

16. 16
How DID I do it?
•Challenges
–Primarily for a LAB
–Finding a good UI for the users
–Emulating as MUCH functionality as possible
•At what point do I “Draw the line”
•Still building some of this

17. 17
How DID I do it?
•Some Zero-Tier Terminology
–Planet
•Zero-Tier Root Servers
–Moon
•User Defined Root Server
–Leaf
•SD-WAN Endpoints
•Controllers

18. 18
How DID I do it?
•ZT Docker image
–Unable to run an ARM image as a VM
•Had to go x86(CHR)
•Sits off to the side of the rest of the routing engine
–Missing some tooling
•jq – Parsing JSON queries from the Mikrotik API
•curl – execute the API queries
–Preinstall my “Moon” files
•Still unsure if I can even do this on ARM_64 MT
–Based on the original image
•https://hub.docker.com/r/zerotier/zerotier

19. 19
How DID I do it?
•Step 1
–Create some new ROOT servers
•At least 2 Recommended
•Tooling is built-in to do this
–https://docs.zerotier.com/zerotier/moons/
•Step 2
–Block access to the planet servers
•IPTABLES rules/firewall rules should be sufficient
•Step 3
–Install the “MOONS” on your client nodes
•Details included in the above

20. 20
How DID I do it?
•Step 4
–Install a Node to use as a controller
•Step 5
–Choose a GUI
•https://github.com/dec0dOS/zero-ui
•https://github.com/key-networks/ztncui
–They have their Pros and Cons
•Step 6
–Setup your networks and Join your clients
–Configure any routing required on your end nodes

21. 21
Should I do it?
•What Problem am I trying to solve?
–Cost?
–Service Availability?
–Splitting services?
–User Self-Management?
•They All have different answers
–Can I do it another way?
–Will a standard VPN do the same thing?

22. 22
Should I do it?
•Use Case #1
MEF-70.1 Page 98

23. 23
Should I do it?
•Use Case #2
MEF-70.1 Page 99

24. 24
Demo Topology

25. 25
Quick 2 Minute Demo

26. 26
Questions?