2023年7月23日发(作者：)

⼀次Flannel和Docker⽹络不通定位问题 查看路由表的配置路由表情况[root@k8s-master ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0 192.168.44.1 0.0.0.0 UG 100 0 0 enp0s310.1.0.0 0.0.0.0 255.255.0.0 U 0 0 0 flannel010.1.19.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0192.168.44.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s3192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr010.1.0.0为flannel0⽹段⽽在这台机器上启动的pod都是在10.1.19.0⽹段的 node的节点路由表[root@node1 flannel]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0 192.168.44.1 0.0.0.0 UG 100 0 0 enp0s310.1.0.0 0.0.0.0 255.255.0.0 U 0 0 0 flannel010.1.28.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0192.168.44.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s3192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr010.1.0.0为flannel0⽹段⽽在这台机器上启动的pod都是在10.1.28.0⽹段的 所有pod的ip地址[root@k8s-master ~]# kubectl get pods -o wideNAME READY STATUS RESTARTS AGE IP NODEhelloworld-service-2437162702-r9v2q 2/2 Running 9 9d 10.1.28.3 node1helloworld-service-v2-2637126738-s284c 2/2 Running 10 9d 10.1.28.4 node1istio-egress-2869428605-2ftgl 1/1 Running 6 13d 10.1.28.6 node1istio-ingress-1286550044-6g3vj 1/1 Running 6 13d 10.1.28.5 node1istio-mixer-765485573-23wc6 1/1 Running 6 13d 10.1.28.7 node1istio-pilot-1495912787-g5r9s 2/2 Running 11 13d 10.1.28.9 node1tool-185907110-ms991 2/2 Running 4 8d 10.1.28.8 node1 正常情况下，ping pod节点的⽹络应该是通的[root@k8s-master ~]# ping 10.1.28.3PING 10.1.28.3 (10.1.28.3) 56(84) bytes of data.64 bytes from 10.1.28.3: icmp_seq=1 ttl=61 time=0.967 ms64 bytes from 10.1.28.3: icmp_seq=2 ttl=61 time=1.88 ms64 bytes from 10.1.28.3: icmp_seq=3 ttl=61 time=0.867 ms64 bytes from 10.1.28.3: icmp_seq=4 ttl=61 time=2.23 ms

整个通讯链路原理及报⽂追踪 整个链路简单的图如下⽐较详细的可以参考下⾯这张

数据从源容器中发出后，经由所在主机的docker0虚拟⽹卡转发到flannel0虚拟⽹卡，这是个P2P的虚拟⽹卡，flanneld服务监听在⽹卡的另外⼀端。

Flannel通过Etcd服务维护了⼀张节点间的路由表。

源主机的flanneld服务将原本的数据内容UDP封装后根据⾃⼰的路由表投递给⽬的节点的flanneld服务，数据到达以后被解包，然后直 接进⼊⽬的节点的flannel0虚拟⽹卡，然后被转发到⽬的主机的docker0虚拟⽹卡，最后就像本机容器通信⼀下的有docker0路由到达⽬标容 器。

所以要定位⽹络的不通就需要⼀步步的看报⽂是在哪处的转发出了问题。源端⽹络⾸先查看发器端的flannel0的地址[root@k8s-master ~]# ifconfigdocker0: flags=4099 mtu 1500 inet 10.1.19.1 netmask 255.255.255.0 broadcast 0.0.0.0 ether 02:42:3a:a6:1d:bb txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0enp0s3: flags=4163 mtu 1500 inet 192.168.44.108 netmask 255.255.255.0 broadcast 192.168.44.255 inet6 fe80::a00:27ff:fee2:ae0a prefixlen 64 scopeid 0x20 ether 08:00:27:e2:ae:0a txqueuelen 1000 (Ethernet) RX packets 20866 bytes 2478600 (2.3 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 21990 bytes 13812121 (13.1 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0flannel0: flags=4305 mtu 1472 inet 10.1.19.0 netmask 255.255.0.0 destination 10.1.19.0 unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 500 (UNSPEC) RX packets 14 bytes 1176 (1.1 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 14 bytes 1176 (1.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 然后运⾏下⾯命令监听从flannel0出去的任何的包tcpdump -i flannel0 -nn host 10.1.19.0同时再找个窗⼝ping pod,这是收到的信息是 ping 10.1.28.3[root@k8s-master ~]# tcpdump -i flannel0 -nn host 10.1.19.0tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on flannel0, link-type RAW (Raw IP), capture size 65535 bytes16:28:43.961488 IP 10.1.19.0 > 10.1.28.3: ICMP echo request, id 4520, seq 1, length 6416:28:43.963340 IP 10.1.28.3 > 10.1.19.0: ICMP echo reply, id 4520, seq 1, length 6416:28:44.962567 IP 10.1.19.0 > 10.1.28.3: ICMP echo request, id 4520, seq 2, length 6416:28:44.963339 IP 10.1.28.3 > 10.1.19.0: ICMP echo reply, id 4520, seq 2, length 6416:28:45.966388 IP 10.1.19.0 > 10.1.28.3: ICMP echo request, id 4520, seq 3, length 6416:28:45.966962 IP 10.1.28.3 > 10.1.19.0: ICMP echo reply, id 4520, seq 3, length 6416:28:46.967629 IP 10.1.19.0 > 10.1.28.3: ICMP echo request, id 4520, seq 4, length 6416:28:46.968486 IP 10.1.28.3 > 10.1.19.0: ICMP echo reply, id 4520, seq 4, length 64可以看到报⽂已经发出，然后看发送端的物理⽹卡enp0s3,继续运⾏ping命令，然后看有没有转发到物理⽹卡的包因为是master节点，所以有很多8080,443端⼝发的包，可以忽略，真实环境中相对⽐较少.核⼼可以看最后为UDP,length 84,属于把ping的包64封装后的包的⼤⼩。[root@k8s-master ~]# tcpdump -i enp0s3 -nn host 192.168.44.109tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on enp0s3, link-type EN10MB (Ethernet), capture size 65535 bytes16:46:59.146611 IP 192.168.44.108.8080 > 192.168.44.109.50060: Flags [P.], seq 1518764712:1518765120, ack 650646529, win 327, options [nop,nop,TS val 4304586 ecr 7794005], length 40816:46:59.146863 IP 192.168.44.108.443 > 192.168.44.109.51564: Flags [P.], seq 474973224:474973663, ack 3595606551, win 248, options [nop,nop,TS val 4304586 ecr 7794006], length 43916:46:59.147013 IP 192.168.44.109.50060 > 192.168.44.108.8080: Flags [.], ack 408, win 1424, options [nop,nop,TS val 7794610 ecr 4304586], length 016:46:59.147301 IP 192.168.44.109.51564 > 192.168.44.108.443: Flags [.], ack 439, win 1407, options [nop,nop,TS val 7794610 ecr 4304586], length 016:46:59.224901 IP 192.168.44.109.5353 > 224.0.0.251.5353: 0*- [0q] 1/0/0 (Cache flush) PTR . (109)16:46:59.259598 IP 192.168.44.109.34266 > 192.168.44.108.443: Flags [P.], seq 3602262654:3602262700, ack 901869271, win 1407, options [nop,nop,TS val 7794724 ecr 4297197], length 4616:46:59.267671 IP 192.168.44.108.8285 > 192.168.44.109.8285: UDP, length 8416:46:59.269133 IP 192.168.44.109.8285 > 192.168.44.108.8285: UDP, length 8416:46:59.270082 IP 192.168.44.108.443 > 192.168.44.109.34266: Flags [P.], seq 1:66, ack 46, win 1432, options [nop,nop,TS val 4304709 ecr 7794724], length 6516:46:59.270419 IP 192.168.44.108.443 > 192.168.44.109.34266: Flags [P.], seq 66:639, ack 46, win 1432, options [nop,nop,TS val 4304709 ecr 7794724], length 57316:46:59.270734 IP 192.168.44.109.34266 > 192.168.44.108.443: Flags [.], ack 66, win 1407, options [nop,nop,TS val 7794735 ecr 4304709], length 016:46:59.271040 IP 192.168.44.109.34266 > 192.168.44.108.443: Flags [.], ack 639, win 1407, options [nop,nop,TS val 7794735 ecr 4304709], length 016:46:59.272370 IP 192.168.44.109.34266 > 192.168.44.108.443: Flags [P.], seq 46:94, ack 639, win 1407, options [nop,nop,TS val 7794736 ecr 4304709], length 4816:46:59.272522 IP 192.168.44.109.34266 > 192.168.44.108.443: Flags [P.], seq 94:667, ack 639, win 1407, options [nop,nop,TS val 7794736 ecr 4304709], length 57316:46:59.272743 IP 192.168.44.109.34266 > 192.168.44.108.443: Flags [P.], seq 667:705, ack 639, win 1407, options [nop,nop,TS val 7794736 ecr 4304709], length 3816:46:59.278885 IP 192.168.44.108.443 > 192.168.44.109.34266: Flags [.], ack 705, win 1432, options [nop,nop,TS val 4304718 ecr 7794736], length 016:46:59.283084 IP 192.168.44.108.443 > 192.168.44.109.34266: Flags [P.], seq 639:681, ack 705, win 1432, options [nop,nop,TS val 4304722 ecr 7794736], length 4216:46:59.283224 IP 192.168.44.108.443 > 192.168.44.109.34266: Flags [P.], seq 681:723, ack 705, win 1432, options [nop,nop,TS val 4304722 ecr 7794736], length 4216:46:59.284143 IP 192.168.44.109.34266 > 192.168.44.108.443: Flags [.], ack 723, win 1407, options [nop,nop,TS val 7794748 ecr 4304722], length 016:46:59.287279 IP 192.168.44.108.8080 > 192.168.44.109.50060: Flags [P.], seq 408:824, ack 1, win 327, options [nop,nop,TS val 4304726 ecr 7794610], length 41616:46:59.287584 IP 192.168.44.109.50060 > 192.168.44.108.8080: Flags [.], ack 824, win 1424, options [nop,nop,TS val 7794751 ecr 4304726], length 0命令确认ping命令的包发到192.168.44.109

⽬标段⽹络再去node1⽬标端，看物理⽹卡的收包情况,源端继续运⾏ping[root@node1 flannel]# tcpdump -i enp0s3 -nn host 192.168.44.108tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on enp0s3, link-type EN10MB (Ethernet), capture size 65535 bytes16:49:04.022476 IP 192.168.44.108.8285 > 192.168.44.109.8285: UDP, length 8416:49:04.022827 IP 192.168.44.109.8285 > 192.168.44.108.8285: UDP, length 8416:49:05.022980 IP 192.168.44.108.8285 > 192.168.44.109.8285: UDP, length 8416:49:05.023425 IP 192.168.44.109.8285 > 192.168.44.108.8285: UDP, length 8416:49:05.273652 IP 192.168.44.108.8080 > 192.168.44.109.50060: Flags [P.], seq 1518824053:1518824479, ack 650646776, win 336, options [nop,nop,TS val 4430711 ecr 7919368], length 42616:49:05.273754 IP 192.168.44.109.50060 > 192.168.44.108.8080: Flags [.], ack 426, win 1424, options [nop,nop,TS val 7920736 ecr 4430711], length 016:49:05.273951 IP 192.168.44.108.443 > 192.168.44.109.51564: Flags [P.], seq 475036916:475037373, ack 3595607190, win 248, options [nop,nop,TS val 4430711 ecr 7919369], length 45716:49:05.274091 IP 192.168.44.109.51564 > 192.168.44.108.443: Flags [.], ack 457, win 1407, options [nop,nop,TS val 7920737 ecr 4430711], length 0发现源端有包过来，正常 在⽬标节点node1上运⾏，10.1.19.0是源端的flannel0地址，正常。[root@node1 flannel]# tcpdump -i flannel0 -nn host 10.1.19.0tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on flannel0, link-type RAW (Raw IP), capture size 65535 bytes16:51:49.795788 IP 10.1.19.0 > 10.1.28.3: ICMP echo request, id 4797, seq 1, length 6416:51:49.795911 IP 10.1.28.3 > 10.1.19.0: ICMP echo reply, id 4797, seq 1, length 6416:51:50.797484 IP 10.1.19.0 > 10.1.28.3: ICMP echo request, id 4797, seq 2, length 6416:51:50.797566 IP 10.1.28.3 > 10.1.19.0: ICMP echo reply, id 4797, seq 2, length 6416:51:51.796934 IP 10.1.19.0 > 10.1.28.3: ICMP echo request, id 4797, seq 3, length 6416:51:51.797024 IP 10.1.28.3 > 10.1.19.0: ICMP echo reply, id 4797, seq 3, length 6416:51:52.800567 IP 10.1.19.0 > 10.1.28.3: ICMP echo request, id 4797, seq 4, length 6416:51:52.800641 IP 10.1.28.3 > 10.1.19.0: ICMP echo reply, id 4797, seq 4, length 64最后看⽬标端docker0有没有报⽂，28.3⽬标pod地址tcpdump -i docker0 -nn host 10.1.28.3

问题定位遇到的问题是⽬标端flannel0上有包发过来，但docker0⽹段没有任何包。所以定位是⽬标段的flannel0->docker0的转发出了问题。通过iptables -nvL 查看现有的iptables规则，发现chain FORWARD链路 policy是DROP,以下命令修改iptables -P FORWARD ACCEPT 另外查宿主机的 ip forward是否有问题sysctl -a | grep ip_forward

源端如何找到⽬标端地址全靠flannel会找etcd的中的数据，然后进⾏路由

华丽的分割线====================================================================================服务启动顺序

Kubernetes启动这些服务的顺序⾮常重要先是flannel Serviceflannel服务启动时主要做了以下⼏步的⼯作：从etcd中获取network的配置信息划分subnet，并在etcd中进⾏注册将⼦⽹信息记录到/run/flannel/中cat /run/flannel/NNEL_NETWORK=10.0.0.0/16FLANNEL_SUBNET=10.0.53.1/24FLANNEL_MTU=1450FLANNEL_IPMASQ=false

之后将会有⼀个脚本将转写成⼀个docker的环境变量⽂件/run/flannel/dockercat /run/flannel/docker

DOCKER_OPT_BIP="--bip=10.0.53.1/24"DOCKER_OPT_IPMASQ="--ip-masq=true"DOCKER_OPT_MTU="--mtu=1450"DOCKER_NETWORK_OPTIONS=" --bip=10.0.53.1/24 --ip-masq=true --mtu=1450 "

然后是Docker服务Docker服务会根据flannel拿到的⽹段然后把pod启动在这些⽹段，这样Kubernetes在寻址pod的时候就会找到相应的宿主机，进⾏通讯。如果Docker服务和Flannel服务没有这种关联关系的化，很可能Docker先⽤原来的ip段启动，⽽这个段并没有写到etcd中，导致寻址失败。这就是在另⼀次定位问题的出错点。

==============================================================验证是否开墙开通nc -u 10.93.0.131 (host B) 8472输⼊字符再host B上，通过tcpdump -i eth0 -nn host hostA来验证是否能收到报⽂