eBPF vs Sidecars by Liz Rice at Isovalent

From its vantage point in the kernel, eBPF provides a platform for building a new generation of infrastructure tools for things like observability, security and networking. These kinds of facilities used to be implemented as libraries, and then in container environments they were often deployed as sidecars. In this talk let's consider why eBPF can offer numerous advantages over these models, particularly when it comes to performance.

1. eBPF vs Sidecars Liz Rice Chief Open Source Officer, Isovalent

2. 👋 Hi, I’m Liz ( she/her ) Chief Open Source Officer at Isovalent Previously chair of CNCF’s Technical Oversight Committee Early career: writing networking code Containers / security / eBPF / cloud native Often found on a bike or playing music

3. What is a sidecar?

4. Sidecar model evolution Python app Go app Go app 2 Go library Go library Python library

5. Sidecar model evolution Pod Pod Pod Python app Go app Go app 2 Go library Go library Python library

6. Sidecar model evolution Pod Pod Pod Python app Go app Go app 2 Sidecar Go library Sidecar Go library Sidecar Go library

7. Sidecar containers Share namespaces / cgroups with application container Allow injecting common tooling into every pod Shares lifecycle with application container

8. Sidecar containers Share namespaces / cgroups with application container Allow injecting common tooling into every pod Shares lifecycle with application container → operational complexity Tooling roll-out requires pod restart Undefined start-up ordering

9. Sidecars → operational complexity concerns “ can be problematic with certain server-speaks-first protocols” “start-up/shut-down race conditions” “adds a lot of complexity and overhead” “overly complex to operate and scale” “really frustrated with the sidecar models” “complexity that causes issues for developers and ops alike”

10. Nathan LeClaire @dotpem

11. eBPF makes the kernel dynamically programmable

12. U serspace kernel P od P od container container container syscalls

13. U serspace kernel P od P od container One kernel per host container container syscalls

14. U serspace kernel One kernel per host syscalls networking access files create containers P od P od container container container

15. kernel Kernel aware of everything on host networking access files create containers U serspace P od P od container container container

16. kernel eBPF programs can be aware of everything networking access files create containers No changes to apps or config needed U serspace P od P od container container container

17. U serspace P od P od app app app kernel sidecar sidecar A sidecar has a view across just one pod

18. U serspace P od P od app app app kernel sidecar sidecar Sidecar created through YAML my-app.yaml containers: name: my-app ... name: my-app-init ... name: my-sidecar ...

19. U serspace P od P od app app app kernel eBPF doesn’t require any app changes my-app.yaml containers: name: my-app ... name: my-app-init ...

20. U serspace P od P od app app app kernel eBPF doesn’t even require any app restarts my-app.yaml containers: name: my-app ... name: my-app-init ...

21. U serspace P od P od app app app kernel eBPF can see ALL activity on the node my-app.yaml containers: name: my-app ... name: my-app-init ... 👿

22. Sidecar containers Shares lifecycle with application container → operational complexity Allow injecting common tooling into every pod But needs changes to pod spec YAML for each instrumented app Share namespaces / cgroups with application container

23. Sidecar containers Shares lifecycle with application container → operational complexity Allow injecting common tooling into every pod But needs changes to pod spec YAML for each instrumented app Share namespaces / cgroups with application container Isolated from other pods …and from other sidecar containers

24. Sidecars → resource usage concerns “ looking for something with smaller footprint ” “too much overhead” “struggling to operate a low cost cluster due to sidecar overhead” “biggest concerns we have are scalability, performance and raw resource consumption, and the added latency and complexity”

25. U serspace kernel Reduce resource usage

26. U serspace kernel U serspace kernel Reduce resource usage maps

27. Sidecars → latency concerns “biggest concerns we have are scalability, performance and raw resource consumption, and the added latency and complexity” “s idecarless and reducing hops approach would reduce latency ” “ we're not hugely fond of the sidecar model and the extra latency & complexity involved ” “ we've avoided service meshes to date due to resource overhead and latency concerns ”

28. The network cost of sidecar proxies u serspace kernel p od app network proxy syscalls

29. The network cost of sidecar proxies u serspace kernel p od app network proxy syscalls socket socket socket tcp/ip tcp/ip tcp/ip networking eth0

30. The network cost of sidecar proxies u serspace kernel p od app network proxy syscalls socket socket socket tcp/ip tcp/ip tcp/ip networking eth0

31. userspace kernel The network cost of sidecar proxies

32. userspace kernel Removing sidecars, retaining proxy features userspace kernel

33. Can we move proxy features into the kernel? userspace kernel userspace kernel app 1 service mesh library tcp/ip networking tcp/ip networking App 1 App 2 a pp 1 a pp 2 s ervice mesh sidecar s ervice mesh sidecar userspace kernel tcp/ip networking a pp 1 a pp 2 L7 proxy s ervice mesh app 2 service mesh library

34. High performance eBPF-based visibility

35. Liz Rice | @lizrice ebpf.io | cilium.io | i sovalent.com Thank you! Let’s connect.