Zero-overhead Container Networking with eBPF and Netkit by Liz Rice

Introducing Netkit: a new eBPF enhancement replacing veth connections in container networking. Say goodbye to the overhead slowing down container apps. With Netkit, container networking now matches the speed of host networking. Fast, efficient, and ready to deploy. #DevOps #eBPF

1. A ScyllaDB Community
Zero-overhead Container Networking
with eBPF and Netkit
Liz Rice
Chief Open Source Officer, Isovalent @ Cisco

2. ?????? Hi, I’m Liz (she/her)
Chief Open Source Officer, Isovalent @ Cisco
■Previously chair of CNCF’s Technical
Oversight Committee
■Early career: writing networking code
■Containers / security / eBPF / cloud
native
■Often found on a bike or playing music

3. Container networking overhead

4. Traditional host networking
Host
app
eth0
network stack
(IP, netfilter,
routing, …)
Network interface card
socket

5. Container networking
Host
Container app
eth0
network stack
(IP, netfilter,
routing, …)
network interface card
veth
veth
virtual
Ethernet
connection
Host’s network
namespace
Container’s network
namespace
network
stack
socket

6. Container networking
Host
Container app
eth0
network stack
(IP, netfilter,
routing, …)
network interface card
veth
veth
virtual
Ethernet
connection
Host’s network
namespace
Container’s network
namespace
network
stack
socket
Traffic passes
through network
stack twice ??????
TCP back
pressure
breakages ??????

7. Use eBPF programs to do
host routing
New eBPF helper functions in 5.10

8. Host routing with eBPF - ingress
Host
Container app
eth0
network stack
(IP, netfilter,
routing, …)
veth
veth
Host’s network
namespace
Container’s network
namespace
network
stack
socket
eBPF
programs
Ingress: bpf_redirect_peer()
Switch packet’s netns✕

9. Host routing with eBPF - egress
Host
Container app
eth0
veth
veth
Host’s network
namespace
Container’s network
namespace
network
stack
socket
eBPF
programs
Egress: bpf_redirect_neigh()
FIB lookup + fill in L2 info
network stack
(IP, netfilter,
routing, …)✕

10. Host routing with eBPF
Host
Container app
eth0
veth
veth
Host’s network
namespace
Container’s network
namespace
network
stack
socket
eBPF
programs
network stack
(IP, netfilter,
routing, …)✕
eBPF
programs

11. So far so good, but still some overhead

12. veth backlog queue

12
Deferral to
ksoftirqd

13. Host routing with eBPF
Host
Container app
eth0
veth
veth
Host’s network
namespace
Container’s network
namespace
network
stack
socket
eBPF
programs
network stack
(IP, netfilter,
routing, …)✕
eBPF
programs
Egress packets
can go on per-CPU
backlog queue ??????
Ingress packets
skip per-CPU
backlog queue

14. tcx - TC express
TC = traffic control subsystem
in kernel network stack

15. eBPF program attachment in TC
Before kernel 6.6

16. eBPF program attachment in TC
Before kernel 6.6 New style: tcx
59 cycles -> 33 cycles to
BPF program entry

17. Replace veth devices
with netkit devices
Minimal eBPF-programmable devices

18. Netkit devices
Host
Container
netkit
primary
netkit
peer
Host’s network
namespace
Container’s network
namespace
eBPF
programs
eBPF
programs
Only primary can manage eBPF
programs

19. Netkit devices
Host
Container app
eth0
netkit
netkit
Host’s network
namespace
Container’s network
namespace
network
stack
socket
eBPF
programs
eBPF
programs
Fast netns
switching and no
queuing for both
ingress and
egress
network stack
(IP, netfilter,
routing, …)✕

20. veth vs netkit: backlog queue

20
Pod with veth:
Pod with
netkit:
Remains in process
context all the way,
leading to better process
scheduler decisions.
Deferral to
ksoftirqd

21. Container networking overhead eliminated!
Throughput as
high as host
baseline

22. Container networking overhead eliminated!
Latency as
low as host
baseline

23. Netkit support now in Cilium

24. Thank you!
Liz Rice
@lizrice
isovalent.com/blog