k8s from kube-proxy and iptables-by Martynas.pdf

k8s from kube-proxy and iptables

1. Liberating K8s from
kube-proxy and iptables
(and netfilter)
Martynas Pumputis, Cilium
(Daniel Borkmann, Thomas Graf, André Martins)

2. # perf top -a -e cycles:k
PerfTop: 16326 irqs/sec (all, 4 CPUs)
-----------------------------------------------------------------------------------
8.79% [kernel] [k] native_sched_clock
4.99% [ip_tables] [k] ipt_do_table
3.09% [e1000e] [k] e1000_irq_enable
2.51% [nf_conntrack] [k] __nf_conntrack_find_get
2.03% [kernel] [k] fib_table_lookup
1.98% [kernel] [k] sched_clock_cpu
1.75% [nf_conntrack] [k] tcp_packet
1.65% [nf_conntrack] [k] nf_conntrack_tuple_taken
[...]
Performance

3. Reliability
May 27,
2018
Root
cause
Aug 5,
2018
Patches
submitte
d
Feb 11,
2019
Patches
merged

4. Reliability
Feb 11,
2019
Patches
merged
Nov 11,
2010
First
occurance of
bug

5. Debuggability
# iptables-save -c
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
[1:10] -A FORWARD -i eth0 -s 172.17.0.0/16 -j DROP

6. Debuggability
* raw
:PR EROUTIN G ACCEP T [4827 4:48 68066 3]
:OUTPUT ACCE PT [46 709:33 5067 71]
COM M IT
* mang le
:PR EROUTIN G ACCEP T [4827 4:48 68066 3]
:IN PUT AC CEPT [ 4820 3:4867 7293]
:FORWA RD ACC EPT [70: 3334 ]
:OUTPUT ACCE PT [46 709:33 5067 71]
:POSTROUTIN G ACCE PT [467 78:3 35100 20]
COM M IT
* nat
:PR EROUTIN G ACCEP T [0:0]
:IN PUT AC CEPT [ 0:0]
:OUTPUT ACCE PT [31 :1905]
:POSTROUTIN G ACCE PT [21: 1305 ]
:D OC KER - [0: 0]
:KUBE- M ARK-D ROP - [ 0:0]
:KUBE- M ARK-M A SQ - [0:0 ]
:KUBE- NODEP ORTS - [0:0 ]
:KUBE- POSTR OUTING - [0 :0]
:KUBE- SEP-AR IYJBM S CT6N PKLC - [ 0:0]
:KUBE- SEP-EV B54 GPOXM 4P4KY H - [ 0:0]
:KUBE- SEP- JN EFD VS5 622RF 3KK - [ 0:0]
:KUBE- SEP-LH V3D TYFO2 UR3 QEF - [ 0:0 ]
:KUBE- SEP-PV CRD UM N ZPYK 3THF - [0:0 ]
:KUBE- SEP-RY4 UH CSD DTR J5BR D - [ 0:0 ]
:KUBE- SEP-YQP47 3NS N3 FT5 3LX - [0 :0]
:KUBE- SERV IC ES - [0:0 ]
:KUBE- SVC -ERIFX ISQEP7F7OF4 - [0 :0]
:KUBE- SVC - JD 5M R3N A4I 4DYORP - [ 0:0]
:KUBE- SVC -NPX 46M 4PTM TK RN6 Y - [0: 0]
:KUBE- SVC -TC OU7JCQXE ZGV UN U - [0:0 ]
-A P RER OUTIN G -m comm ent --comme nt " kubern etes se rvice port als " -j KUBE -SERV ICES
-A P RER OUTIN G -m ad drt ype --ds t- type LOCAL - j D OCKER
-A OUT PUT -m com ment --comm ent " kuber netes s ervice por tals " -j KUB E-SERV ICE S
-A OUT PUT ! -d 127.0 .0.0/8 - m addr type -- ds t-t ype LOC AL -j D OC KER
-A P OSTR OUTING -m com ment --comm ent " kuber netes pos tr outing rules " -j KUBE-POSTR OUTING
-A P OSTR OUTING -s 172. 17.0.0/ 16 ! - o docker 0 -j M AS QUERA DE
-A P OSTR OUTING -s 172. 18.0.0/ 16 ! - o docker _ gw bridg e -j M ASQUE RAD E
-A D OCKER -i docker0 -j RETURN
-A D OCKER -i docker_ gw br idge -j R ETUR N
-A K UB E-M AR K-DR OP - j M ARK --s et-xm ark 0x8 000/0x 8000
-A K UB E-M AR K-M ASQ -j M AR K -- set -xmar k 0x4000 /0x400 0
-A K UB E-POSTROUTIN G - m comm en t -- com ment "kub ernete s s er vice t raffic re quiring SN AT" -m mar k - -mar k 0x400 0/0x40 00 -j M AS QUERA DE
-A K UB E-SEP-A RIYJB M SCT6N PKLC - s 10.217 .0.224/ 32 -j KUBE- M ARK-M A SQ
-A K UB E-SEP-A RIYJB M SCT6N PKLC - p tcp -m tcp -j DN AT --t o -des tina tion 10 .217.0.2 24:91 53
-A K UB E-SEP-E VB 54GPOXM 4 P4KYH -s 1 0.217. 0.71/32 -j KUB E-M A RK-M A SQ
-A K UB E-SEP-E VB 54GPOXM 4 P4KYH -p tcp - m tcp - j DN AT --to -des tin ation 10 .2 17.0.71 :915 3
-A K UB E-SEP- JNEF DV S5622R F3KK - s 10.217 .0 .224/ 32 -j KUBE-M ARK-M A SQ
-A K UB E-SEP- JNEF DV S5622R F3KK - p tcp -m tcp -j DN AT --t o -des tina tion 10 .217.0.2 24:53
-A K UB E-SEP-LH V3 DTYFO2UR 3QEF -s 192.1 68.1.12 5/32 - j KUBE -M AR K- M ASQ
-A K UB E-SEP-LH V3 DTYFO2UR 3QEF -p t cp - m t cp - j DN AT --t o -des t in ation 1 92.168 .1.125: 6443
-A K UB E-SEP-P VCR DUM NZ PYK3TH F -s 10.21 7.0.224/ 32 -j KUBE- M AR K-M ASQ
-A K UB E-SEP-P VCR DUM NZ PYK3TH F -p udp -m u dp -j D NAT -- to -d est inat ion 10.217 .0.224 :53
-A K UB E-SEP-RY4UH CS DD TRJ5B RD -s 10.21 7.0.71/ 32 -j KUBE- M ARK-M A SQ
-A K UB E-SEP-RY4UH CS DD TRJ5B RD -p t cp - m t cp - j DN AT --t o -des t in ation 1 0.217.0 .71:53
-A K UB E-SEP-YQP473N SN 3FT53LX - s 10 .217.0 .71/32 -j KUB E-M A RK-M AS Q
-A K UB E-SEP-YQP473N SN 3FT53LX - p udp - m udp -j DN AT --t o -des tin ation 1 0.217.0 .71:53
-A K UB E-SERV ICE S ! -s 10 .217.0.0/ 16 -d 10.96.0.1 0/32 - p tcp -m comm ent --com ment "kube -sy st em/kube -dns :dn s-t cp clus ter I P" -m tcp -- dport 53 -j KUBE- M AR K-M ASQ
-A K UB E-SERV ICE S -d 10 .96.0.10 /32 -p tcp -m com ment --comm ent " kube-s ys tem /kube-d ns :dns -tcp clu st er IP" -m t cp --dpo rt 5 3 -j KUB E-S VC- ERIFXI SQE P7F7OF4
-A K UB E-SERV ICE S ! -s 10 .217.0.0/ 16 -d 10.96.0.1 0/32 - p tcp -m comm ent --com ment "kube -sy st em/kube -dns :m etrics clus ter I P" -m tcp -- dport 9153 -j KUB E-M AR K-M ASQ
-A K UB E-SERV ICE S -d 10 .96.0.10 /32 -p tcp -m com ment --comm ent " kube-s ys tem /kube-d ns :met rics clu st er IP" -m t cp --dpo rt 9 153 -j K UBE -SVC - JD 5M R3 NA 4I4DYORP
-A K UB E-SERV ICE S ! -s 10 .217.0.0/ 16 -d 10.96.0.1 /32 - p tcp -m com men t --com ment "def au lt/kube rnet es: http s clu st er IP" -m t cp --dpo rt 4 43 -j KUBE- MA RK-M A SQ
-A K UB E-SERV ICE S -d 10 .96.0.1/ 32 -p t cp -m com ment - -comme nt " default /kuber netes :ht tps clus ter IP" -m tcp -- dpor t 443 - j KUB E-SV C-N PX4 6M 4PTM TKR N6Y
-A K UB E-SERV ICE S ! -s 10 .217.0.0/ 16 -d 10.96.0.1 0/32 - p udp - m comm ent -- commen t " kub e-s ys tem/ku be-dns :d ns clus t er IP" -m u dp --dp ort 53 -j KUBE- MA RK-M A SQ
-A K UB E-SERV ICE S -d 10 .96.0.10 /32 -p udp -m co mment --comm ent "kube- sys t em /kube- dns :dns clus ter IP" -m udp - -dpor t 53 - j KUB E-SV C-TCOU7 JCQXEZ GV UN U
-A K UB E-SERV ICE S -m comm ent --comm en t " kuberne tes s er vice nodepor ts ; N OTE : this m us t be t he las t ru le i n this chain" -m a ddrt ype --d st -type LOCA L -j KUB E-N OD EPORTS
-A K UB E-SV C-ER IFXISQEP7F7 OF4 -m s tatis t ic --mode random --pr obability 0.5000 00000 00 -j KUBE- SEP- JN EFD VS5 622RF 3KK
-A K UB E-SV C-ER IFXISQEP7F7 OF4 -j KUB E-SEP- RY4UH CSD D TRJ5B RD
-A K UB E-SV C- JD 5M R3 NA 4I4DYORP - m s tat ist ic --mod e random --pr obabilit y 0.5000 00000 00 -j KUBE- SEP-A RIYJBM S CT6N PKLC
-A K UB E-SV C- JD 5M R3 NA 4I4DYORP - j KUBE -SEP- EVB5 4GP OXM 4P 4KYH
-A K UB E-SV C-N PX46M 4PTM TKRN 6Y -j K UBE -SEP-LH V3D TYFO2UR 3QE F
-A K UB E-SV C-TCOU7JC QX EZGVUNU -m s ta tis tic --m ode rand om -- probab ili ty 0.50 000000 000 -j KUB E-SEP- PVCR DUM NZ PYK3TH F
-A K UB E-SV C-TCOU7JC QX EZGVUNU -j KUBE-SE P-Y QP47 3NSN 3FT53LX
COM M IT
* filter
:IN PUT AC CEPT [ 2938 :62362 0]
:FORWA RD DR OP [ 0:0]
:OUTPUT ACCE PT [28 93:671 491]
:D OC KER - [0: 0]
:D OC KER-I SOL ATION- STAGE-1 - [ 0:0]
:D OC KER-I SOL ATION- STAGE-2 - [ 0:0]
:D OC KER-USER - [0 :0]
:KUBE- EXTERN AL- SERVIC ES - [0:0]
:KUBE- FIREWA LL - [0:0 ]
:KUBE- FORWA RD - [0 :0]
:KUBE- SERV IC ES - [0:0 ]
-A I NPUT - m connt rack --ct st ate N EW -m comm en t --com ment "kub ernet es s er vice po rtals " -j KUBE-S ERVIC ES
-A I NPUT - m connt rack --ct st ate N EW -m comm en t --com ment "kub ernet es ext ernally- vis ible ser vice porta ls" -j KUBE- EXTER NAL- SERVI CES
-A I NPUT - j KUB E-FIRE WALL
-A F ORWARD -m com ment --comm ent " kubern etes for war ding r ules " -j KUB E-FORWAR D
-A F ORWARD -m conn tra ck - -cts tate NEW -m com ment - -comme nt " kubern etes se rvice port als" -j K UBE -SERV ICES
-A F ORWARD -j D OC KER- USER
-A F ORWARD -j D OC KER- ISOL ATION -STAGE -1
-A F ORWARD -o docker0 -m connt rack --cts t ate RE L ATED ,ESTAB LISH ED -j AC CEPT
-A F ORWARD -o docker0 -j DOCK ER
-A F ORWARD -i docker0 ! -o docker0 -j ACC EPT
-A F ORWARD -i docker0 - o docke r0 -j AC CE PT
-A F ORWARD -o docker_ gw br idge -m co nntr ack --cts tat e REL ATE D,ES TA BLISH ED - j ACC EPT
-A F ORWARD -o docker_ gw br idge -j D OCKER
-A F ORWARD -i docker_ gw br idge ! - o docke r_ gw bridg e -j ACC EPT
-A F ORWARD -i docker_ gw br idge -o d ocker _gw b ridge - j D R OP
-A OUT PUT -m con ntr ack - -ct s tat e N EW -m com ment --comm ent " ku bern et es s ervice port als " -j K UBE -SERV ICES
-A OUT PUT -j KUBE- FI REWA LL
-A D OCKER-ISOL ATION -STAGE-1 -i docker0 ! -o docker 0 -j D OCKER-ISOL ATION -STAGE-2
-A D OCKER-ISOL ATION -STAGE-1 -i docker_ gw bridg e ! -o d ocker_ g wb ridge - j DOCKE R-ISOL ATION -STAGE-2
-A D OCKER-ISOL ATION -STAGE-1 -j RETURN
-A D OCKER-ISOL ATION -STAGE-2 -o docker0 -j D ROP
-A D OCKER-ISOL ATION -STAGE-2 -o docker_ gw brid ge -j DR OP
-A D OCKER-ISOL ATION -STAGE-2 -j RETURN
-A D OCKER-US ER -j R ETUR N
-A K UB E-FIREWALL -m comm ent --comm en t " kuberne tes firew all for dropp ing ma rked packets " -m m ar k --m ark 0x80 00/0 x8 000 - j D ROP
-A K UB E-FORWAR D -m connt rack --cts t ate INVALID -j DR OP
-A K UB E-FORWAR D -m comm en t --com ment "kub ernet es f or w ardin g rules " -m m ark -- mark 0 x4 000/ 0x4000 - j ACC EPT
-A K UB E-FORWAR D -s 10 .217.0. 0/16 -m comm ent --com ment "kube rnet es f or w arding co nntr ack pod s ource ru le" - m conntr ack --cts t at e RE LATED,ES TABLISH ED -j ACC EPT
-A K UB E-FORWAR D -d 10.21 7.0.0/1 6 -m comm ent --comm en t " kuberne tes forw ar ding connt rack pod d est ination rule" -m connt rack --ct st ate R EL ATED ,ESTAB LI SHE D -j AC CEP T
-A K UB E-SERV ICE S -d 10 .99.38.1 55/32 -p tcp - m comme nt -- com ment "de fault/ nginx-59 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .96.61.2 52/32 -p tcp - m comme nt -- com ment "de fault/ nginx-64 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .104.16 6.10/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-6 7: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .98.85.4 1/32 - p tcp -m co mmen t --com ment " defa ult/n ginx-9: h as n o endpoin ts " - m tcp - -dport 80 - j REJEC T --r eje ct -w ith icm p-port -un reachable
-A K UB E-SERV ICE S -d 10 .97.138 .144/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-1 7: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .106.49 .80/32 -p tcp - m comme nt -- com ment "de fault/ nginx-37 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .104.16 4.205/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 5: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .104.25 .150/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-1 9: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .106.23 4.213/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 88: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .109.20 9.136/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 33: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .106.19 6.105/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 49: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .111.10 1.6/32 -p tcp - m comme nt -- com ment "de fault/ nginx-53 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .110.22 6.230/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 79: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .98.99.1 36/32 -p tcp - m comme nt -- com ment "de fault/ nginx-6: has no endp oi nts " -m t cp --dpor t 80 -j REJE CT --r eject -w ith icm p-por t-u nreachable
-A K UB E-SERV ICE S -d 10 .99.75.2 33/32 -p tcp - m comme nt -- com ment "de fault/ nginx-7: has no endp oi nts " -m t cp --dpor t 80 -j REJE CT --r eject -w ith icm p-por t-u nreachable
-A K UB E-SERV ICE S -d 10 .108.41 .202/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-1 4: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .97.36.2 49/32 -p tcp - m comme nt -- com ment "de fault/ nginx-99 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .98.213 .37/32 -p tcp - m comme nt -- com ment "de fault/ nginx-77 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .107.22 9.31/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-9 2: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .98.64.2 51/32 -p tcp - m comme nt -- com ment "de fault/ nginx-16 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .101.88 .159/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-3 1: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .105.71 .74/32 -p tcp - m comme nt -- com ment "de fault/ nginx-41 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .108.92 .226/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-6 3: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .109.25 2.234/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 18: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .104.11 8.66/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-3 0: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .106.22 4.55/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-8 3: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .109.16 .199/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-1 00: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .109.23 1.213/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 61: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .98.27.2 50/32 -p tcp - m comme nt -- com ment "de fault/ nginx-95 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .105.42 .108/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-1 2: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .99.35.2 36/32 -p tcp - m comme nt -- com ment "de fault/ nginx-20 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .111.42 .123/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-2 1: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .99.47.2 25/32 -p tcp - m comme nt -- com ment "de fault/ nginx-22 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .104.18 4.242/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 51: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .98.77.9 3/32 - p tcp -m co mmen t --com ment " defa ult/n ginx-68: has no endp oi nts " -m t cp --dpor t 80 -j REJE CT --r eject -w ith icm p-por t-u nreachable
-A K UB E-SERV ICE S -d 10 .110.16 9.113/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 72: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .100.23 1.169/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 90: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .96.58.5 1/32 - p tcp -m co mmen t --com ment " defa ult/n ginx-4: h as n o endpoin ts " - m tcp - -dport 80 - j REJEC T --r eje ct -w ith icm p-port -un reachable
-A K UB E-SERV ICE S -d 10 .101.13 2.61/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-2 5: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .100.64 .242/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-3 9: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .111.15 4.81/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-5 0: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .100.17 9.151/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 96: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .111.69 .30/32 -p tcp - m comme nt -- com ment "de fault/ nginx-35 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .96.35.2 12/32 -p tcp - m comme nt -- com ment "de fault/ nginx-38 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .96.61.2 6/32 - p tcp -m co mmen t --com ment " defa ult/n ginx-84: has no endp oi nts " -m t cp --dpor t 80 -j REJE CT --r eject -w ith icm p-por t-u nreachable
-A K UB E-SERV ICE S -d 10 .96.229 .244/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-8 7: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .104.24 7.138/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 66: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .96.214 .153/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-1 1: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .102.20 8.205/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 55: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .106.35 .32/32 -p tcp - m comme nt -- com ment "de fault/ nginx-58 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .107.17 4.56/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-6 5: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .98.142 .83/32 -p tcp - m comme nt -- com ment "de fault/ nginx-2: has no endp oi nts " -m t cp --dpor t 80 -j REJE CT --r eject -w ith icm p-por t-u nreachable
-A K UB E-SERV ICE S -d 10 .106.24 8.222/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 15: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .98.202 .86/32 -p tcp - m comme nt -- com ment "de fault/ nginx-34 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .96.57.2 13/32 -p tcp - m comme nt -- com ment "de fault/ nginx-71 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .98.33.1 99/32 -p tcp - m comme nt -- com ment "de fault/ nginx-69 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .101.93 .81/32 -p tcp - m comme nt -- com ment "de fault/ nginx-75 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .99.199 .226/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-7 8: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .103.12 2.17/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-1 0: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .103.19 4.216/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 27: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .97.117 .130/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-3 2: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .98.254 .254/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-5 6: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .100.16 4.89/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-2 9: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .106.18 7.33/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-4 2: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .111.68 .111/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-4 4: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .97.54.1 35/32 -p tcp - m comme nt -- com ment "de fault/ nginx-46 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .106.12 8.46/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-1 3: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .108.22 3.155/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 26: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .108.10 1.195/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 62: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .102.12 4.200/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 73: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .101.14 1.155/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 93: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .96.141 .192/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-7 0: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .110.19 8.145/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 80: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .104.23 7.179/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 24: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .106.19 8.6/32 -p tcp - m comme nt -- com ment "de fault/ nginx-36 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .110.24 7.41/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-4 0: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .111.21 9.198/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 60: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .105.21 4.185/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 52: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .106.56 .25/32 -p tcp - m comme nt -- com ment "de fault/ nginx-54 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .100.14 4.20/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-8 6: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .97.106 .133/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-8 9: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .97.137 .184/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-2 3: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .103.24 3.253/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 28: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .100.99 .151/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-4 3: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .96.231 .60/32 -p tcp - m comme nt -- com ment "de fault/ nginx-47 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .104.17 3.153/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 98: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .100.19 4.184/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 94: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .99.198 .225/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-9 7: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .108.15 4.23/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-1 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .107.29 .154/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-4 8: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .110.22 4.213/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 85: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .110.14 6.9/32 -p tcp - m comme nt -- com ment "de fault/ nginx-91 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .100.17 4.231/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 74: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .101.24 1.20/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-7 6: has no en dpoint s" -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unr ea cha ble
-A K UB E-SERV ICE S -d 10 .96.49.1 15/32 -p tcp - m comme nt -- com ment "de fault/ nginx-81 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .100.19 7.189/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 82: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .105.11 9.26/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-3 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .109.23 7.26/3 2 -p t cp -m comm ent - -co mmen t "d efault/ nginx-8 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
-A K UB E-SERV ICE S -d 10 .105.13 2.182/ 32 -p t cp -m comm ent --comme nt " default /ngin x- 45: h as no endpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .99.220 .77/32 -p tcp - m comme nt -- com ment "de fault/ nginx-57 : has no end points " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w it h icmp-p ort- unreachab le
COM M IT
https://www.reddit.com/r/networkingmemes/comments/8u7jyz/
container_networking/
* raw
:PR EROUTIN G AC CEP T [ 4827 4:486 80663 ]
:OUTPUT ACCE PT [46 709:33 50677 1]
COM M IT
* mang le
:PR EROUTIN G AC CEP T [ 4827 4:486 80663 ]
:IN PUT AC CEPT [ 4820 3:4867 7293]
:FORWAR D ACCE PT [70: 3334]
:OUTPUT ACCE PT [46 709:33 50677 1]
:POSTROUTIN G ACCE PT [467 78:33 51002 0]
COM M IT
* nat
:PR EROUTIN G AC CEP T [ 0:0]
:IN PUT AC CEPT [ 0:0]
:OUTPUT ACCE PT [31 :1905]
:POSTROUTIN G ACCE PT [21: 1305]
:D OC KER - [0: 0]
:KUBE- MA RK-D ROP - [ 0:0]
:KUBE- MA RK-M A SQ - [0:0 ]
:KUBE- NODEP ORTS - [0:0 ]
:KUBE- POSTR OUTING - [0 :0]
:KUBE- SEP-AR IYJBM S CT6N PKLC - [ 0:0]
:KUBE- SEP-EV B54 GPOX M 4P4KY H - [ 0:0]
:KUBE- SEP- JN EFD VS5 622RF 3KK - [ 0:0]
:KUBE- SEP-LH V3D TYFO2 UR3 QEF - [ 0:0 ]
:KUBE- SEP-PV CRD UM N ZPYK 3THF - [0:0 ]
:KUBE- SEP-RY4UH CSD DTR J5BR D - [ 0:0 ]
:KUBE- SEP-YQP47 3NS N3 FT5 3LX - [0 :0]
:KUBE- SERVIC ES - [ 0:0]
:KUBE- SVC -ERIFX ISQEP7F7OF4 - [0 :0]
:KUBE- SVC - JD 5M R3N A4I 4DYOR P - [ 0:0]
:KUBE- SVC -NPX 46M 4PTM TK RN6 Y - [0: 0]
:KUBE- SVC -TC OU7JCQXE ZGV UN U - [0:0 ]
-A P REROUTIN G -m comm ent - -comme nt " ku bern et es se rvice port als" -j KUBE -SERV ICES
-A P REROUTIN G -m add rt yp e --ds t- type LOCAL -j D OCKER
-A OUT PUT -m com ment --comm ent " kuber netes s ervice por tals " -j KUB E-SERV ICE S
-A OUT PUT ! -d 127.0 .0.0/8 - m addr type -- ds t-t ype LOC AL -j D OC KER
-A P OSTR OUTING -m com ment --comm ent " kuber netes pos tr outing rules " -j KUB E-POSTR OUTIN G
-A P OSTR OUTING -s 172. 17.0.0/ 16 ! -o docker 0 -j M AS QUERA DE
-A P OSTR OUTING -s 172. 18.0.0/ 16 ! -o docker _ gw bridge -j M ASQUE RAD E
-A D OCKER -i docker0 - j RETURN
-A D OCKER -i docker_ gw br idge -j R ETUR N
-A K UB E-M AR K-DR OP -j M ARK - -s et-xm ark 0x8 000/0x 8000
-A K UB E-M AR K-M ASQ -j M AR K -- set -xmar k 0x4000 /0x400 0
-A K UB E-POS TROUTIN G - m commen t -- com ment "kub er nete s s ervice t raffic re quiring SN AT" -m mar k - -mar k 0x4000 /0x400 0 -j M AS QUERA DE
-A K UB E-SEP-A RIYJB M SCT6N PKLC - s 1 0.217.0 .224/3 2 -j KUBE-M ARK-M A SQ
-A K UB E-SEP-A RIYJB M SCT6N PKLC - p tcp -m tcp -j DN AT --to -des tina tion 10 .217.0.22 4:91 53
-A K UB E-SEP-E VB 54GPOXM 4 P4 KYH -s 1 0.217. 0.71/32 -j KUB E-M A RK-M AS Q
-A K UB E-SEP-E VB 54GPOXM 4 P4 KYH -p tcp - m tcp - j D N AT --to -des tin at ion 10.2 17.0.71 :9153
-A K UB E-SEP- JNEF DV S5622R F3KK - s 1 0.217.0 .224/3 2 -j KUBE-M ARK-M A SQ
-A K UB E-SEP- JNEF DV S5622R F3KK - p tcp -m tcp -j DN AT --to -des tina tion 10 .2 17.0.22 4:53
-A K UB E-SEP-LH V3D TYFO2UR 3QEF -s 192.16 8.1.125 /32 - j K UBE -M AR K- M ASQ
-A K UB E-SEP-LH V3D TYFO2UR 3QEF -p t cp - m t cp - j DN AT --t o -des tin ation 1 92.168 .1.125: 6443
-A K UB E-SEP-P VCR DUM NZ PYK3TH F -s 10.217 .0.224/ 32 -j KUBE- M ARK-M A SQ
-A K UB E-SEP-P VCR DUM NZ PYK3TH F -p udp -m ud p -j D NAT -- to -d est inat ion 10.217 .0.224 :53
-A K UB E-SEP-RY4UH CSD D TRJ5B RD -s 10.217 .0.71/3 2 -j KUBE-M ARK-M A SQ
-A K UB E-SEP-RY4UH CSD D TRJ5B RD -p t cp - m t cp - j DN AT --t o -des tin ation 1 0.217.0 .7 1:53
-A K UB E-SEP-YQP473N SN 3FT53LX - s 10 .217.0 .71/32 -j KUB E-M A RK-M ASQ
-A K UB E-SEP-YQP473N SN 3FT53LX - p udp - m udp -j DN AT --t o -des tin ation 1 0.217.0.7 1:53
-A K UB E-SERV ICE S ! -s 10.2 17.0.0/ 16 -d 1 0.96.0.1 0/32 - p tcp -m comm en t --com ment "kube -sy st em/kube -dns :dn s-t cp clus ter I P" -m tcp -- dport 53 -j KUBE- M ARK-M A SQ
-A K UB E-SERV ICE S -d 10 .96.0.10/ 32 -p tcp -m com ment --comm ent " kube-s ys tem /kube-dn s :dns -tcp clus t er IP" -m t cp --dpo rt 5 3 -j KUB E-S VC- ERIFXI SQE P7 F7OF4
-A K UB E-SERV ICE S ! -s 10.2 17.0.0/ 16 -d 1 0.96.0.1 0/32 - p tcp -m comm en t --com ment "kube -sy st em/kube -dns :m et rics clus ter I P" -m tcp -- dport 9153 - j KUB E-M AR K-M ASQ
-A K UB E-SERV ICE S -d 10 .96.0.10/ 32 -p tcp -m com ment --comm ent " kube-s ys tem /kube-dn s :metr ics clus t er IP" -m t cp --dpo rt 9 153 -j K UBE- SVC - JD 5M R3 NA 4I4DYORP
-A K UB E-SERV ICE S ! -s 10.2 17.0.0/ 16 -d 1 0.96.0.1 /32 -p tcp -m com ment --comm ent "defau lt/kube rnet es : http s clus t er IP" -m t cp --dpo rt 4 43 -j KUBE-M A RK-M A SQ
-A K UB E-SERV ICE S -d 10 .96.0.1/ 32 -p t cp -m com ment - -comme nt " default /kuber netes :ht tps clus ter IP" -m tcp -- dpor t 443 - j KUB E-SV C-N PX4 6M 4PTM TKRN 6Y
-A K UB E-SERV ICE S ! -s 10.2 17.0.0/ 16 -d 1 0.96.0.1 0/32 - p udp - m comm ent -- com ment " kub e-s yst em/kub e-dns :d ns clus t er IP" -m u dp --dp ort 53 -j KUBE-M A RK-M A SQ
-A K UB E-SERV ICE S -d 10 .96.0.10/ 32 -p udp -m co mment --comm ent " kube-s ys tem /kube- dns :dns clus ter IP" -m udp - -dpor t 53 - j KUB E-SV C-TCOU7JC QX EZGVUN U
-A K UB E-SERV ICE S -m comm ent - -commen t " kuberne tes ser vice nodepor ts ; N OTE : this mu s t be th e las t ru le i n this chain" -m a ddrt ype --d st -type LOCA L - j KUB E-N OD EPORTS
-A K UB E-SVC -ER IFXISQEP7F7OF4 -m s tatis t ic --mode random --pr obability 0.5000 00000 00 -j KUBE- SEP- JN EFD VS56 22RF 3KK
-A K UB E-SVC -ER IFXISQEP7F7OF4 -j KUBE -SEP- RY4UH CSD D TRJ5B RD
-A K UB E-SVC - JD 5M R3 NA 4I4DYORP - m s tat ist ic - -mod e r andom --pr obabilit y 0 .50000 00000 0 -j KUBE-S EP-A RIYJBM SC T6N PKLC
-A K UB E-SVC - JD 5M R3 NA 4I4DYORP - j K UBE -SEP-E VB5 4GP OXM 4P 4KYH
-A K UB E-SVC -N PX46M 4 PTM TKRN 6Y -j K UBE- SEP-LH V3D TYFO2UR3 QEF
-A K UB E-SVC -TCOU7JC QX EZGVUNU -m s ta tis tic --m ode rand om --p robabili ty 0.50 000000 000 -j KUB E-SEP- PV CR DUM NZ PYK3TH F
-A K UB E-SVC -TCOU7JC QX EZGVUNU -j KUB E-SE P-Y QP473 NSN 3FT53LX
COM M IT
* filter
:IN PUT AC CEPT [ 2938 :62362 0]
:FORWAR D D R OP [0 :0]
:OUTPUT ACCE PT [28 93:671 491]
:D OC KER - [0: 0]
:D OC KER-I SOL ATION- STAGE -1 - [ 0:0]
:D OC KER-I SOL ATION- STAGE -2 - [ 0:0]
:D OC KER-USER - [0: 0]
:KUBE- EXTERN AL- SERVIC ES - [ 0:0]
:KUBE- FIR EWA LL - [ 0:0]
:KUBE- FORWA RD - [0 :0]
:KUBE- SERVIC ES - [ 0:0]
-A I NPUT - m connt rack --cts t ate N EW -m commen t --com ment "kub er netes s er vice po rtals " -j KUBE-SE RVIC ES
-A I NPUT - m connt rack --cts t ate N EW -m commen t --com ment "kub er netes ext ernally-vis ible s er vice p orta ls" -j KUBE- EXTER NAL-S ERVI CES
-A I NPUT - j KUB E-FIRE WALL
-A F ORWA RD -m com ment --comm ent " kubern etes forw ar ding r ules " -j KUBE -FORWAR D
-A F ORWA RD -m conn tra ck - -ct s tate N EW -m com ment - -comme nt " ku bern et es se rvice port als" -j K UBE -SERV ICES
-A F ORWA RD -j D OC KER- USER
-A F ORWA RD -j D OC KER- ISOL ATION -STAGE -1
-A F ORWA RD -o docker0 -m connt rack --cts t ate RE L ATED ,ESTAB LISH ED -j AC CEPT
-A F ORWA RD -o docker0 -j DOCK ER
-A F ORWA RD -i docker0 ! -o docker0 - j ACC EPT
-A F ORWA RD -i docker0 - o docke r0 -j AC CE PT
-A F ORWA RD -o docker_ gw br idge -m co nntr ack - -cts tat e REL ATE D,ESTA BLISH ED - j ACC EPT
-A F ORWA RD -o docker_ gw br idge -j D OCKER
-A F ORWA RD -i docker_ gw br idge ! - o docke r_ gw bridg e -j ACC EPT
-A F ORWA RD -i docker_ gw br idge -o docker _gw br idge -j D ROP
-A OUT PUT -m con ntr ack -- ct s tat e N EW -m comm ent --comm en t " ku bernet es s er vice port als " -j KUBE- SERV ICES
-A OUT PUT -j KUBE-FI REWA LL
-A D OCKER-ISOL ATION -STAGE-1 -i docker 0 ! - o docker 0 -j D OCKER- ISOL ATION -STAGE-2
-A D OCKER-ISOL ATION -STAGE-1 -i docker _ gw bridg e ! -o d ocker_ gw b ridge - j D OCKER-ISOL ATION -STAGE-2
-A D OCKER-ISOL ATION -STAGE-1 -j RETURN
-A D OCKER-ISOL ATION -STAGE-2 -o docker 0 -j D ROP
-A D OCKER-ISOL ATION -STAGE-2 -o docker _ gw brid ge -j DR OP
-A D OCKER-ISOL ATION -STAGE-2 -j RETURN
-A D OCKER-US ER -j R ETUR N
-A K UB E-FIREWALL -m comm ent - -commen t " kuberne tes firew all for dropp ing ma rked packets " -m mar k --m ark 0x80 00/0x8 000 -j D ROP
-A K UB E-FORWAR D -m connt rack --cts t at e INVALID - j DR OP
-A K UB E-FORWAR D -m commen t --com ment "kub er netes for w ardin g rules " -m m ark --m ark 0x4 000/0 x4000 - j ACC EPT
-A K UB E-FORWAR D -s 10 .217.0. 0/16 -m comm en t --com ment "kube rnetes for w arding co nntr ack p od s ource ru le" -m conntr ack --cts tat e REL ATE D,ES TA BLISH ED -j ACC EPT
-A K UB E-FORWAR D -d 10.21 7.0.0/1 6 -m comm ent - -commen t " kuberne tes forw ar ding connt rack pod d es t ination rule" -m connt rack --ct st ate R EL ATED ,ESTAB LI SHE D -j AC CEP T
-A K UB E-SERV ICE S -d 10 .99.38.15 5/32 -p tcp - m comme nt -- com ment "de fault/n ginx-59 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .96.61.25 2/32 -p tcp - m comme nt -- com ment "de fault/n ginx-64 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .104.16 6.10/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-6 7: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .98.85.41 /32 - p tcp -m co mmen t --com ment " defa ult/ng inx-9: h as n o endpoin ts " - m tcp - -dport 80 - j R EJEC T --reje ct -w ith icmp -port -un reachable
-A K UB E-SERV ICE S -d 10 .97.138 .1 44/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-1 7: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .106.49 .8 0/32 -p tcp - m comme nt -- com ment "de fault/n ginx-37 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .104.16 4.205/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 5: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .104.25 .1 50/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-1 9: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .106.23 4.213/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 88: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .109.20 9.136/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 33: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .106.19 6.105/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 49: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .111.10 1.6/32 -p tcp - m comme nt -- com ment "de fault/n ginx-53 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .110.22 6.230/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 79: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .98.99.13 6/32 -p tcp - m comme nt -- com ment "de fault/n ginx-6: has no endpoi nts " - m tcp - -dpor t 80 - j REJE CT --r eject -w ith icm p-por t-u nreachable
-A K UB E-SERV ICE S -d 10 .99.75.23 3/32 -p tcp - m comme nt -- com ment "de fault/n ginx-7: has no endpoi nts " - m tcp - -dpor t 80 - j REJE CT --r eject -w ith icm p-por t-u nreachable
-A K UB E-SERV ICE S -d 10 .108.41 .2 02/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-1 4: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .97.36.24 9/32 -p tcp - m comme nt -- com ment "de fault/n ginx-99 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .98.213 .3 7/32 -p tcp - m comme nt -- com ment "de fault/n ginx-77 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .107.22 9.31/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-9 2: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .98.64.25 1/32 -p tcp - m comme nt -- com ment "de fault/n ginx-16 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .101.88 .1 59/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-3 1: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .105.71 .7 4/32 -p tcp - m comme nt -- com ment "de fault/n ginx-41 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .108.92 .2 26/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-6 3: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .109.25 2.234/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 18: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .104.11 8.66/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-3 0: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .106.22 4.55/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-8 3: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .109.16 .1 99/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-1 00: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .109.23 1.213/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 61: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .98.27.25 0/32 -p tcp - m comme nt -- com ment "de fault/n ginx-95 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .105.42 .1 08/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-1 2: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .99.35.23 6/32 -p tcp - m comme nt -- com ment "de fault/n ginx-20 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .111.42 .1 23/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-2 1: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .99.47.22 5/32 -p tcp - m comme nt -- com ment "de fault/n ginx-22 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .104.18 4.242/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 51: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .98.77.93 /32 - p tcp -m co mmen t --com ment " defa ult/ng inx-68: has no endpoi nts " - m tcp - -dpor t 80 - j REJE CT --r eject -w ith icm p-por t-u nreachable
-A K UB E-SERV ICE S -d 10 .110.16 9.113/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 72: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .100.23 1.169/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 90: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .96.58.51 /32 - p tcp -m co mmen t --com ment " defa ult/ng inx-4: h as n o endpoin ts " - m tcp - -dport 80 - j R EJEC T --reje ct -w ith icmp -port -un reachable
-A K UB E-SERV ICE S -d 10 .101.13 2.61/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-2 5: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .100.64 .2 42/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-3 9: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .111.15 4.81/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-5 0: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .100.17 9.151/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 96: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .111.69 .3 0/32 -p tcp - m comme nt -- com ment "de fault/n ginx-35 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .96.35.21 2/32 -p tcp - m comme nt -- com ment "de fault/n ginx-38 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .96.61.26 /32 - p tcp -m co mmen t --com ment " defa ult/ng inx-84: has no endpoi nts " - m tcp - -dpor t 80 - j REJE CT --r eject -w ith icm p-por t-u nreachable
-A K UB E-SERV ICE S -d 10 .96.229 .2 44/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-8 7: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .104.24 7.138/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 66: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .96.214 .1 53/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-1 1: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .102.20 8.205/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 55: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .106.35 .3 2/32 -p tcp - m comme nt -- com ment "de fault/n ginx-58 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .107.17 4.56/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-6 5: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .98.142 .8 3/32 -p tcp - m comme nt -- com ment "de fault/n ginx-2: has no endpoi nts " - m tcp - -dpor t 80 - j REJE CT --r eject -w ith icm p-por t-u nreachable
-A K UB E-SERV ICE S -d 10 .106.24 8.222/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 15: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .98.202 .8 6/32 -p tcp - m comme nt -- com ment "de fault/n ginx-34 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .96.57.21 3/32 -p tcp - m comme nt -- com ment "de fault/n ginx-71 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .98.33.19 9/32 -p tcp - m comme nt -- com ment "de fault/n ginx-69 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .101.93 .8 1/32 -p tcp - m comme nt -- com ment "de fault/n ginx-75 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .99.199 .2 26/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-7 8: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .103.12 2.17/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-1 0: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .103.19 4.216/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 27: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .97.117 .1 30/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-3 2: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .98.254 .2 54/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-5 6: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .100.16 4.89/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-2 9: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .106.18 7.33/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-4 2: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .111.68 .1 11/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-4 4: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .97.54.13 5/32 -p tcp - m comme nt -- com ment "de fault/n ginx-46 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .106.12 8.46/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-1 3: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .108.22 3.155/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 26: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .108.10 1.195/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 62: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .102.12 4.200/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 73: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .101.14 1.155/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 93: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .96.141 .1 92/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-7 0: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .110.19 8.145/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 80: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .104.23 7.179/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 24: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .106.19 8.6/32 -p tcp - m comme nt -- com ment "de fault/n ginx-36 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .110.24 7.41/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-4 0: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .111.21 9.198/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 60: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .105.21 4.185/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 52: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .106.56 .2 5/32 -p tcp - m comme nt -- com ment "de fault/n ginx-54 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .100.14 4.20/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-8 6: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .97.106 .1 33/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-8 9: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .97.137 .1 84/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-2 3: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .103.24 3.253/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 28: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .100.99 .1 51/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-4 3: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .96.231 .6 0/32 -p tcp - m comme nt -- com ment "de fault/n ginx-47 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .104.17 3.153/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 98: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .100.19 4.184/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 94: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .99.198 .2 25/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-9 7: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .108.15 4.23/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-1 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .107.29 .1 54/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-4 8: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .110.22 4.213/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 85: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .110.14 6.9/32 -p tcp - m comme nt -- com ment "de fault/n ginx-91 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .100.17 4.231/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 74: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .101.24 1.20/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-7 6: has no end points " -m tcp --dp ort 80 -j R EJECT - -reject- wit h icmp- port -unrea cha ble
-A K UB E-SERV ICE S -d 10 .96.49.11 5/32 -p tcp - m comme nt -- com ment "de fault/n ginx-81 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .100.19 7.189/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 82: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .105.11 9.26/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-3 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .109.23 7.26/3 2 -p tcp - m comm ent -- co mmen t "d efault/ nginx-8 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
-A K UB E-SERV ICE S -d 10 .105.13 2.182/ 32 -p t cp -m comm ent - -comme nt " default /nginx- 45: has no en dpoint s " -m tcp --d port 80 -j R EJECT --reject -wi th icmp -port -unr each able
-A K UB E-SERV ICE S -d 10 .99.220 .7 7/32 -p tcp - m comme nt -- com ment "de fault/n ginx-57 : has no endp oints " -m t cp --dpo rt 8 0 -j RE JECT --r eject-w ith icm p-p or t- unreachab le
COM M IT

7. CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?
curid=122201
alloc
skb
Packet flow
TC
ingress
raw
PREROUTI
NG
conntrack
mangle
PREROUTIN
G
nat
PREROUTI
NG
FIB
lookup
mangle
FORWARD
filter
FORWARD
mangle
POSTROUTI
NG
nat
POSTROUTIN
G
TC
egress
hos
t
po
d
lxc0
eth
0

8. $ kubectl get svc nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
nginx ClusterIP 3.3.3.3 <none> 80/TCP
$ kubectl get endpoints nginx
NAME ENDPOINTS
nginx 1.1.1.1:80, 1.1.2.2:80
ClusterIP with iptables
-t nat -A PREROUTING -m conntrack --ctstate NEW -j KUBE-SERVICES
-A KUBE-SERVICES ! -s 1.1.0.0/16 -d 3.3.3.3/32 -p tcp -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 3.3.3.3/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-NGINX
-A KUBE-SVC-NGINX -m statistic --mode random --probability 0.50 -j KUBE-SEP-NGINX1
-A KUBE-SVC-NGINX -j KUBE-SEP-NGINX2
-A KUBE-SEP-NGINX1 -s 1.1.1.1/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-NGINX1 -p tcp -m tcp -j DNAT --to-destination 1.1.1.1:80
-A KUBE-SEP-NGINX2 -s 1.1.2.2/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-NGINX2 -p tcp -m tcp -j DNAT --to-destination 1.1.2.2:80
nat
PREROUTI
NG

9. CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?
curid=122201
alloc
skb
Packet flow
TC
ingress
raw
PREROUTI
NG
conntrack
mangle
PREROUTIN
G
nat
PREROUTI
NG
FIB
lookup
mangle
FORWARD
filter
FORWARD
mangle
POSTROUTI
NG
nat
POSTROUTIN
G
TC
egress
hos
t
po
d
lxc0
eth
0

10. userspac
e
kernel
JIT
native
code
eth0
eBPF
verifier
bpf(BPF_PROG_LOAD, …)
eBPF
loader
SEC(“to_netdev”)
int handle(struct sk_buff *skb)
{
…
if (tcp->dport == 80)
redirect(lxc0);
return DROP_PACKET;
}

foo.o
clang -target bpf
[...]
agent BPF
map
s
lxc0

11. CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?
curid=122201
alloc
skb
Packet flow
TC
ingress
TC
egress
hos
t
po
d
lxc0
eth
0

12. 268 contributors (Jan 2016 to Nov
2019):
➢443 Daniel Borkmann (Cilium;
maintainer)
➢242 Alexei Starovoitov (Facebook;
maintainer)
➢210 Jakub Kicinski (Netronome)
➢195 Andrii Nakryiko (Facebook)
➢161 Yonghong Song (Facebook)
➢151 Stanislav Fomichev (Google)
➢145 Quentin Monnet (Netronome)
➢144 Martin KaFai Lau (Facebook)
➢139 John Fastabend (Cilium)
➢118 Jesper Dangaard Brouer (Red Hat)
➢[...]
Users:

13. $ kubectl -n kube-system delete ds kube-proxy

14. kube-proxy
1. ClusterIP
- In-cluster access via virtual IP
eth
0
1.1.3.1
client
eth
0
1.1.1.2
nginx
Node A
client
Cluster
IP
eth
0
10.0.0.
1
client
NodePo
rt
client
LoadBala
ncer
Extern
alIP
Node B
2. NodePort
- Access from outside / inside via
node IP + port
3. ExternalIP
- Access from outside via external IP
4. LoadBalancer
- Access from outside via external LB
NodePo
rt

15. ClusterIP (pod to pod) in Cilium
Cilium eBPF datapath
eth
0
1.1.3.1
clien
t
lxc
0
eBPF SVC hash map
SVC IP Port NR => ID EID Endpoint IP
Port
---------------------------------------
---
3.3.3.3 80 1 => 1 4 1.1.1.1
80
3.3.3.3 80 2 => 1 5 1.1.1.2
80
eBPF conntrack LRU map
srcIP sPort dstIP dPort Type => EID|
SVCID
----------------------------------------------
---
1.1.3.1 4321 3.3.3.3 80 SVC => 4
1.1.3.1 4321 1.1.1.1 80 Egress =>
1
eth
0
1.1.1.1
nginx
lxc
0
3.3.3.3:80
(ClusterIP)
Node A
Node
B1.Lookup dst in SVC
map
2.If found:
a.Select EP
b.DNAT
c.Create SVC CT
d.Create Egress
CT1.Lookup Egress CT
2.If found:
a.Rev-DNAT
xlation
b.Redirect to
lxc0
eth0eth0

16. Cilium service maps
kube-
apiserver
eBPF SVC hash map
SVC IP Port NR => ID EID Endpoint IP
Port
-----------------------------------------
-
3.3.3.3 80 1 => 1 4 1.1.1.1
80
3.3.3.3 80 2 => 1 5 1.1.1.2
80
bpf_map_update_element(.
..)
apiVersion: v1
kind:
Endpoints
metadata:
name: nginx
subsets:
- addresses:
- ip: 1.1.1.1
ports:
- port: 80
protocol:
TCP
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
selector:
app: nginx
ports:
- protocol:
TCP
port: 80
clusterIP:
3.3.3.3

17. TCP
ClusterIP (host or pod to pod) in Cilium
eth
0
1.1.1.1
nginx
lxc
0
3.3.3.3:80
(ClusterIP)
clien
t
import “net/http”
func main() {
r, err :=
http.Get("3.3.3.3")
...
}
kern
el
connect()
UDP
1.Lookup dst in SVC
map
2.If found:
a.Change dst
addr and port in
socket

18. NodePort with service endpoint on remote
node in Cilium
eth
0
192.168.0.
1
eth
0
1.1.2.1
redis
lxc
0
Node A
eth
0
192.168.0.2
eth
0
1.1.1.1
nginx
lxc
0
Node
B
10.100.1.1:60000 ->
192.168.0.1:31000
clien
t
1.SVC lookup & DNAT
2.Is endpoint remote?
2.1.eBPF SNAT
2.2.Redirect
192.168.0.1:60000 -> 1.1.1.1:80

19. eth
0
192.168.0.
1
eth
0
1.1.2.1
redis
lxc
0
Node A
eth
0
192.168.0.2
eth
0
1.1.1.1
nginx
lxc
0
Node
B
192.168.0.1:31000 ->
10.100.1.1:60000
clien
t
1.1.1.1:80 -> 192.168.0.1:33000
1.rev-SNAT xlation
2.rev-DNAT xlation
3.Redirect
NodePort with service endpoint on remote
node in Cilium

20. NodePort externalTrafficPolicy=Local
eth
0
192.168.0.
1
eth
0
1.1.2.1
redis
lxc
0
Node A
eth
0
192.168.0.2
eth
0
1.1.1.1
nginx
lxc
0
Node
B
clien
t
10.100.1.1:60000 ->
192.168.0.1:31000

21. NodePort (DSR) in Cilium
eth
0
192.168.0.
1
eth
0
1.1.2.1
redis
lxc
0
Node A
eth
0
192.168.0.2
eth
0
1.1.1.1
nginx
lxc
0
Node
B
10.100.1.1:60000 ->
192.168.0.1:31000
clien
t
10.100.1.1:60000 -> 1.1.1.1:80
1.SVC lookup & DNAT
2.Is endpoint remote?
2.1.Append SVC addr into IP
hdr
2.2.Redirect

22. NodePort (DSR) in Cilium
eth
0
192.168.0.
1
eth
0
1.1.2.1
redis
lxc
0
Node A
eth
0
192.168.0.2
eth
0
1.1.1.1
nginx
lxc
0
Node
B
clien
t
192.168.0.1:31000 -> 10.100.1.1:60000
1.rev-DNAT xlation
2.Redirect

23. Performance (lower is better)

24. Performance (lower is better)

25. Summary
Performance
-Better performance and latency over kube-proxy (ipvs and
iptables)
-Fast service updates
Reliability
-Less LOC in datapath
-No need to wait for a new kernel release to fix a bug
Debuggability
-Better tooling for introspection and troubleshooting
Compatibility
-No more exec iptables
Customization
-Ability to change LB behaviour

26. Want to liberate yourself from
kube-proxy?
Come to our booth (S93)!
https://cilium.link/kubeproxy-free
https://github.com/cilium/cilium

27. Compatibility

28. UDP
ClusterIP (host to pod)
eth
0
1.1.1.1
nginx
lxc
0
3.3.3.3:80
(ClusterIP)
clien
t
import “net/http”
func main() {
r, err :=
http.Get("nginx")
...
}
kern
el
sendmsg()
recvmsg()
TCP
1.Lookup dst in SVC map
2.If found:
a.Change dst addr
and port in
socket
b.Create rev NAT
entry
1.Lookup src in rev NAT
map
2.If found:
a.Change src
addr and port

29. 10.100.1.1:60000 ->
192.168.0.1:31000
192.168.0.1
eth
0
1.1.1.1
nginx
lxc
0
Node
A
clien
t
1.SVC lookup & DNAT
2.Is endpoint local?
2.1.Redirect to lxc0
1.rev-DNAT
xlation
2.Redirect to
eth0
NodePort with service endpoint on local node
in Cilium
eth
0