Illustrated_guid_to_kubernetes_networking.pdf

kubernetes technology

1. Google Cloud Platform
An Illustrated Guide to
Kubernetes Networking
Tim Hockin <[email protected]>
Senior Staff Software Engineer
@thockin

2. Google Cloud Platform
Layer 2: ethernet

3. Google Cloud Platform
node-a
node-c
node-b
node-d
192.168.1.1/16
11:22:33:44:55:01



192.168.1.2/16
01:23:45:67:89:02


192.168.1.3/16
11:22:33:44:55:03


192.168.1.4/16
01:23:45:67:89:04


L2
switch

4. Google Cloud Platform
node-a
node-c
node-b
node-d
L2
to: 192.168.1.3
from: 192.168.1.1
GET / 192.168.1.1/16
11:22:33:44:55:01



192.168.1.2/16
01:23:45:67:89:02


192.168.1.3/16
11:22:33:44:55:03


192.168.1.4/16
01:23:45:67:89:04

5. Google Cloud Platform
node-d
node-bnode-a
node-c
L2
to: <broadcast>
from: 11:22:33:44:55:01
who has 192.168.1.3?
to: 192.168.1.3
from: 192.168.1.1
GET / 192.168.1.1/16
11:22:33:44:55:01



192.168.1.2/16
01:23:45:67:89:02


192.168.1.3/16
11:22:33:44:55:03


192.168.1.4/16
01:23:45:67:89:04


“ARP request”

6. Google Cloud Platform
node-a
node-c
node-b
node-d
L2
to: 192.168.1.3
from: 192.168.1.1
GET / 192.168.1.1/16
11:22:33:44:55:01



192.168.1.2/16
01:23:45:67:89:02


192.168.1.3/16
11:22:33:44:55:03


192.168.1.4/16
01:23:45:67:89:04


to: 11:22:33:44:55:01
from: 11:22:33:44:55:03
I have 192.168.1.3
“ARP response”

7. Google Cloud Platform
node-a
node-c
node-b
node-d
L2
to: 192.168.1.3
via: 11:22:33:44:55:03
from: 192.168.1.1
GET /

192.168.1.1/16
11:22:33:44:55:01



192.168.1.2/16
01:23:45:67:89:02


192.168.1.3/16
11:22:33:44:55:03


192.168.1.4/16
01:23:45:67:89:04

8. Google Cloud Platform
node-a
root netns
eth0: 192.168.1.1/16
11:22:33:44:55:01



L2 with containers
cbr0: 10.0.1.1/24




ctr-1
eth0: 10.0.1.2/24



ctr-2
eth0: 10.0.1.3/24



ctr-3
eth0: 10.0.1.4/24

9. Google Cloud Platform
node-a
192.168.1.1/16
11:22:33:44:55:01

node-c
node-b
node-d
192.168.1.2/16
01:23:45:67:89:02


192.168.1.3/16
11:22:33:44:55:03


192.168.1.4/16
01:23:45:67:89:04


L2
ctr-1 10.0.1.2
aa:bb:cc:dd:e1:01
ctr-2 10.0.3.2
aa:bb:cc:dd:e3:02

10. Google Cloud Platform
node-a
192.168.1.1/16
11:22:33:44:55:01

node-c
node-b
node-d
192.168.1.2/16
01:23:45:67:89:02


192.168.1.3/16
11:22:33:44:55:03


192.168.1.4/16
01:23:45:67:89:04


L2
ctr-1 10.0.1.2
aa:bb:cc:dd:e1:01
ctr-2 10.0.3.2
aa:bb:cc:dd:e3:02
to: 10.0.3.2
from: 10.0.1.2
GET /

11. Google Cloud Platform
node-a
192.168.1.1/16
11:22:33:44:55:01

node-c
node-b
node-d
192.168.1.2/16
01:23:45:67:89:02


192.168.1.3/16
11:22:33:44:55:03


192.168.1.4/16
01:23:45:67:89:04


L2
ctr-1 10.0.1.2
aa:bb:cc:dd:e1:01
ctr-2 10.0.3.2
aa:bb:cc:dd:e3:02
to: 10.0.3.2
from: 10.0.1.2
GET /
to: <broadcast>
from: aa:bb:cc:dd:e1:01
who has 10.0.3.2?
“ARP request”

12. Google Cloud Platform
node-a
192.168.1.1/16
11:22:33:44:55:01

node-c
node-b
node-d
192.168.1.2/16
01:23:45:67:89:02


192.168.1.3/16
11:22:33:44:55:03


192.168.1.4/16
01:23:45:67:89:04


L2
ctr-1 10.0.1.2
aa:bb:cc:dd:e1:01
ctr-2 10.0.3.2
aa:bb:cc:dd:e3:02
to: 10.0.3.2
from: 10.0.1.2
GET /
to: aa:bb:cc:dd:e1:01
from: 11:22:33:44:55:03
I have 10.0.3.2
“proxy ARP
response”

13. Google Cloud Platform
node-a
192.168.1.1/16
11:22:33:44:55:01

node-c
node-b
node-d
192.168.1.2/16
01:23:45:67:89:02


192.168.1.3/16
11:22:33:44:55:03


192.168.1.4/16
01:23:45:67:89:04


L2
ctr-1 10.0.1.2
aa:bb:cc:dd:e1:01
ctr-2 10.0.3.2
aa:bb:cc:dd:e3:02
to: 10.0.3.2
via: 11:22:33:44:55:03
from: 10.0.1.2
GET /

14. Google Cloud Platform
Layer 3 - IP

15. Google Cloud Platform
node-a
node-c
node-b
node-d
192.168.1.1/32


192.168.1.2/32
192.168.1.3/32 192.168.1.4/32
L3
gateway

16. Google Cloud Platform
node-a
node-c
node-b
node-d
192.168.1.1/32


192.168.1.2/32
192.168.1.3/32 192.168.1.4/32
L3
to: 192.168.1.3
from: 192.168.1.1
GET /

17. Google Cloud Platform
node-a
node-c
node-b
node-d
192.168.1.1/32


192.168.1.2/32
192.168.1.3/32 192.168.1.4/32
L3
to: 192.168.1.3
from: 192.168.1.1
GET /

18. Google Cloud Platform
node-a
node-c
node-b
node-d
192.168.1.1/32


192.168.1.2/32
192.168.1.3/32 192.168.1.4/32
L3
to: 192.168.1.3
from: 192.168.1.1
GET /
routing decision,
static or learned
(e.g. BGP)

19. Google Cloud Platform
node-a
192.168.1.1/32

node-c
node-b
node-d
192.168.1.2/32


192.168.1.3/32


192.168.1.4/32


L3
ctr-1 10.0.1.2
ctr-2 10.0.3.2

20. Google Cloud Platform
node-a
192.168.1.1/32

node-c
node-b
node-d
192.168.1.2/32


192.168.1.3/32


192.168.1.4/32


L3
ctr-1 10.0.1.2
ctr-2 10.0.3.2
to: 10.0.3.2
from: 10.0.1.2
GET /

21. Google Cloud Platform
node-a
192.168.1.1/32

node-c
node-b
node-d
192.168.1.2/32


192.168.1.3/32


192.168.1.4/32


L3
ctr-1 10.0.1.2
ctr-2 10.0.3.2
to: 10.0.3.2
from: 10.0.1.2
GET /

22. Google Cloud Platform
node-a
192.168.1.1/32

node-c
node-b
node-d
192.168.1.2/32


192.168.1.3/32


192.168.1.4/32


L3
ctr-1 10.0.1.2
ctr-2 10.0.3.2
to: 10.0.3.2
from: 10.0.1.2
GET /
routing decision,
static or learned
(e.g. BGP)

23. Google Cloud Platform
node-a
192.168.1.1/32

node-c
node-b
node-d
192.168.1.2/32


192.168.1.3/32


192.168.1.4/32


L3
ctr-1 10.0.1.2
ctr-2 10.0.3.2
to: 10.0.3.2
from: 10.0.1.2
GET /

24. Google Cloud Platform
Overlays


Q: When should I use an overlay?
A: When nothing else works, or when you have
specific reasons to want it (e.g. the added value
of management)

25. Google Cloud Platform
node-a
root netns
eth0: 192.168.1.1/16



Overlay (e.g. flannel, weave)

cbr0: 10.0.1.1/24




ctr-1
eth0: 10.0.1.2/24



ctr-2
eth0: 10.0.1.3/24



ctr-3
eth0: 10.0.1.4/24



flannel0: 10.0.1.254/16

26. Google Cloud Platform
node-a
root netns
eth0: 192.168.1.1/16



cbr0: 10.0.1.1/24




ctr-1
eth0: 10.0.1.2/24



ctr-2
eth0: 10.0.1.3/24



ctr-3
eth0: 10.0.1.4/24



flannel0: 10.0.1.254/16




to: 10.0.3.2
from: 10.0.1.2
GET /
Overlay (e.g. flannel, weave)

27. Google Cloud Platform
node-a
root netns
eth0: 192.168.1.1/16



cbr0: 10.0.1.1/24




ctr-1
eth0: 10.0.1.2/24



ctr-2
eth0: 10.0.1.3/24



ctr-3
eth0: 10.0.1.4/24



flannel0: 10.0.1.254/16




to: 10.0.3.2
from: 10.0.1.2
GET /
Overlay (e.g. flannel, weave)

28. Google Cloud Platform
node-a
root netns
eth0: 192.168.1.1/16



cbr0: 10.0.1.1/24




ctr-1
eth0: 10.0.1.2/24



ctr-2
eth0: 10.0.1.3/24



ctr-3
eth0: 10.0.1.4/24



flannel0: 10.0.1.254/16




to: 10.0.3.2
from: 10.0.1.2
GET /
Overlay (e.g. flannel, weave)

29. Google Cloud Platform
node-a
root netns
eth0: 192.168.1.1/16



cbr0: 10.0.1.1/24




ctr-1
eth0: 10.0.1.2/24



ctr-2
eth0: 10.0.1.3/24



ctr-3
eth0: 10.0.1.4/24



flannel0: 10.0.1.254/16




to: 192.168.1.3
from: 192.168.1.1
encap:
to: 10.0.3.2
from: 10.0.1.2
GET /
Overlay (e.g. flannel, weave)

30. Google Cloud Platform
node-a
192.168.1.1/16

node-c
node-b
node-d
192.168.1.2/16


192.168.1.3/16


192.168.1.4/16


ctr-1 10.0.1.2
ctr-2 10.0.3.2
Overlay (e.g. flannel, weave)

31. Google Cloud Platform
node-c
root netns
eth0: 192.168.1.3/16


cbr0: 10.0.3.1/24




ctr-4
eth0: 10.0.3.2/24



ctr-5
eth0: 10.0.3.3/24



ctr-6
eth0: 10.0.3.4/24



flannel0: 10.0.3.254/16




to: 192.168.1.3
from: 192.168.1.1
encap:
to: 10.0.3.2
from: 10.0.1.2
GET /
Overlay (e.g. flannel, weave)

32. Google Cloud Platform
node-c
root netns
eth0: 192.168.1.3/16


cbr0: 10.0.3.1/24




ctr-4
eth0: 10.0.3.2/24



ctr-5
eth0: 10.0.3.3/24



ctr-6
eth0: 10.0.3.4/24



flannel0: 10.0.3.254/16




to: 10.0.3.2
from: 10.0.1.2
GET /
Overlay (e.g. flannel, weave)

33. Google Cloud Platform
node-c
root netns
eth0: 192.168.1.3/16


cbr0: 10.0.3.1/24




ctr-4
eth0: 10.0.3.2/24



ctr-5
eth0: 10.0.3.3/24



ctr-6
eth0: 10.0.3.4/24



flannel0: 10.0.3.254/16




to: 10.0.3.2
from: 10.0.1.2
GET /
Overlay (e.g. flannel, weave)

34. Google Cloud Platform
Overlays - the hard part

35. Google Cloud Platform
node-a
192.168.1.1/16

node-c
non-node
node-d
192.168.1.2/16


192.168.1.3/16


192.168.1.4/16


ctr-1 10.0.1.2
ctr-2 10.0.3.2
Overlay (e.g. flannel, weave)

36. Google Cloud Platform
node-a
192.168.1.1/16

node-c
non-node
node-d
192.168.1.2/16


192.168.1.3/16


192.168.1.4/16


ctr-1 10.0.1.2
ctr-2 10.0.3.2
to: 10.0.3.2
from: 192.168.1.2
GET /
Overlay (e.g. flannel, weave)

37. Google Cloud Platform
node-a
192.168.1.1/16

node-c
non-node
node-d
192.168.1.2/16


192.168.1.3/16


192.168.1.4/16


ctr-1 10.0.1.2
ctr-2 10.0.3.2
to: 10.0.3.2
from: 192.168.1.2
GET /
Overlay (e.g. flannel, weave)

38. Google Cloud Platform
node-a
192.168.1.1/16

node-c
non-node
node-d
192.168.1.2/16


192.168.1.3/16


192.168.1.4/16


ctr-1 10.0.1.2
ctr-2 10.0.3.2
to: 10.0.3.2
from: 192.168.1.2
GET /
?!?!
Overlay (e.g. flannel, weave)

39. Google Cloud Platform
We need a bridge between the
physical and overlay networks...

40. Google Cloud Platform
We need a bridge between the
physical and overlay networks...
•could: route to nodes
•could: route to 1 or more bridge machines
•could: run flannel on client machines

41. Google Cloud Platform
We need a bridge between the
physical and overlay networks...
•could: route to nodes
•could: route to 1 or more bridge machines
•could: run flannel on client machines
•see “When should I use an overlay?”