dialing dial unix /var/run/antrea/cni.sock: connect: connection refused
Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "63a01c2405a404eb56920f2d7af34f19ed88fef0519e76cb4e493fa3f0c4bba9" network for pod "web-nginx-5f769fdc6-dlnqq":
networkPlugin cni failed to set up pod "web-nginx-5f769fdc6-dlnqq_default" network: rpc error:
code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused",
failed to clean up sandbox container "63a01c2405a404eb56920f2d7af34f19ed88fef0519e76cb4e493fa3f0c4bba9" network for pod "web-nginx-5f769fdc6-dlnqq": networkPlugin cni failed to teardown
pod "web-nginx-5f769fdc6-dlnqq_default" network: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused"]
apiVersion: apps/v1 kind: Deployment metadata: name: web-nginx spec: selector: matchLabels: app: web-nginx replicas: 2 template: metadata: labels: app: web-nginx spec: affinity: #pod 反亲和性, 打散 web-nginx 各个副本 podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: web operator: In values: - nginx topologyKey: "kubernetes.io/hostname" containers: - image: nginx imagePullPolicy: IfNotPresent name: web2-worker ports: - containerPort: 8087 protocol: TCP
root@ubuntu:~# kubectl describe pod web-nginx-5f769fdc6-dlnqq Name: web-nginx-5f769fdc6-dlnqq Namespace: default Priority: 0 Node: cloud/10.10.16.47 Start Time: Fri, 18 Jun 2021 10:03:43 +0800 Labels: app=web-nginx pod-template-hash=5f769fdc6 Annotations: <none> Status: Pending IP: IPs: <none> Controlled By: ReplicaSet/web-nginx-5f769fdc6 Containers: web2-worker: Container ID: Image: nginx Image ID: Port: 8087/TCP Host Port: 0/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-ckv9x (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: default-token-ckv9x: Type: Secret (a volume populated by a Secret) SecretName: default-token-ckv9x Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned default/web-nginx-5f769fdc6-dlnqq to cloud Warning FailedCreatePodSandBox 51s kubelet, cloud Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "63a01c2405a404eb56920f2d7af34f19ed88fef0519e76cb4e493fa3f0c4bba9" network for pod "web-nginx-5f769fdc6-dlnqq": networkPlugin cni failed to set up pod "web-nginx-5f769fdc6-dlnqq_default" network: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused", failed to clean up sandbox container "63a01c2405a404eb56920f2d7af34f19ed88fef0519e76cb4e493fa3f0c4bba9" network for pod "web-nginx-5f769fdc6-dlnqq": networkPlugin cni failed to teardown pod "web-nginx-5f769fdc6-dlnqq_default" network: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused"] Normal SandboxChanged 2s (x5 over 51s) kubelet, cloud Pod sandbox changed, it will be killed and re-created.
root@cloud:~# journalctl -u kubelet -f -- Logs begin at Tue 2020-10-20 19:26:58 CST. -- Jun 18 10:09:23 cloud kubelet[406675]: E0618 10:09:23.235161 406675 remote_runtime.go:144] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod "web-nginx-5f769fdc6-tt8mf_default" network: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused"" podSandboxID="3eca7a32f1c2fb9d72ad58ab7dae887b4c6db5bf7892809a2d3e6897c48655b8" Jun 18 10:09:23 cloud kubelet[406675]: E0618 10:09:23.235212 406675 kuberuntime_manager.go:958] "Failed to stop sandbox" podSandboxID={Type:docker ID:3eca7a32f1c2fb9d72ad58ab7dae887b4c6db5bf7892809a2d3e6897c48655b8} Jun 18 10:09:23 cloud kubelet[406675]: E0618 10:09:23.235301 406675 kuberuntime_manager.go:729] "killPodWithSyncResult failed" err="failed to "KillPodSandbox" for "b045617d-721d-477f-8db4-62c4fdc0c358" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"web-nginx-5f769fdc6-tt8mf_default\" network: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused\""" Jun 18 10:09:23 cloud kubelet[406675]: E0618 10:09:23.235369 406675 pod_workers.go:190] "Error syncing pod, skipping" err="failed to "KillPodSandbox" for "b045617d-721d-477f-8db4-62c4fdc0c358" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"web-nginx-5f769fdc6-tt8mf_default\" network: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused\""" pod="default/web-nginx-5f769fdc6-tt8mf" podUID=b045617d-721d-477f-8db4-62c4fdc0c358 Jun 18 10:09:28 cloud kubelet[406675]: I0618 10:09:28.217152 406675 cni.go:333] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container "63a01c2405a404eb56920f2d7af34f19ed88fef0519e76cb4e493fa3f0c4bba9"" Jun 18 10:09:28 cloud kubelet[406675]: E0618 10:09:28.233226 406675 cni.go:380] "Error deleting pod from network" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused"" pod="default/web-nginx-5f769fdc6-dlnqq" podSandboxID={Type:docker ID:63a01c2405a404eb56920f2d7af34f19ed88fef0519e76cb4e493fa3f0c4bba9} podNetnsPath="" networkType="antrea" networkName="antrea" Jun 18 10:09:28 cloud kubelet[406675]: E0618 10:09:28.233972 406675 remote_runtime.go:144] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod "web-nginx-5f769fdc6-dlnqq_default" network: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused"" podSandboxID="63a01c2405a404eb56920f2d7af34f19ed88fef0519e76cb4e493fa3f0c4bba9" Jun 18 10:09:28 cloud kubelet[406675]: E0618 10:09:28.234021 406675 kuberuntime_manager.go:958] "Failed to stop sandbox" podSandboxID={Type:docker ID:63a01c2405a404eb56920f2d7af34f19ed88fef0519e76cb4e493fa3f0c4bba9} Jun 18 10:09:28 cloud kubelet[406675]: E0618 10:09:28.234103 406675 kuberuntime_manager.go:729] "killPodWithSyncResult failed" err="failed to "KillPodSandbox" for "b44537e7-292f-481d-b347-f6df12f1e53a" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"web-nginx-5f769fdc6-dlnqq_default\" network: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused\""" Jun 18 10:09:28 cloud kubelet[406675]: E0618 10:09:28.234183 406675 pod_workers.go:190] "Error syncing pod, skipping" err="failed to "KillPodSandbox" for "b44537e7-292f-481d-b347-f6df12f1e53a" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"web-nginx-5f769fdc6-dlnqq_default\" network: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused\""" pod="default/web-nginx-5f769fdc6-dlnqq" podUID=b44537e7-292f-481d-b347-f6df12f1e53a Jun 18 10:09:35 cloud kubelet[406675]: I0618 10:09:35.217071 406675 cni.go:333] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container "3eca7a32f1c2fb9d72ad58ab7dae887b4c6db5bf7892809a2d3e6897c48655b8"" Jun 18 10:09:35 cloud kubelet[406675]: E0618 10:09:35.232950 406675 cni.go:380] "Error deleting pod from network" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused"" pod="default/web-nginx-5f769fdc6-tt8mf" podSandboxID={Type:docker ID:3eca7a32f1c2fb9d72ad58ab7dae887b4c6db5bf7892809a2d3e6897c48655b8} podNetnsPath="" networkType="antrea" networkName="antrea" Jun 18 10:09:35 cloud kubelet[406675]: E0618 10:09:35.233568 406675 remote_runtime.go:144] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod "web-nginx-5f769fdc6-tt8mf_default" network: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused"" podSandboxID="3eca7a32f1c2fb9d72ad58ab7dae887b4c6db5bf7892809a2d3e6897c48655b8" Jun 18 10:09:35 cloud kubelet[406675]: E0618 10:09:35.233611 406675 kuberuntime_manager.go:958] "Failed to stop sandbox" podSandboxID={Type:docker ID:3eca7a32f1c2fb9d72ad58ab7dae887b4c6db5bf7892809a2d3e6897c48655b8} Jun 18 10:09:35 cloud kubelet[406675]: E0618 10:09:35.234523 406675 kuberuntime_manager.go:729] "killPodWithSyncResult failed" err="failed to "KillPodSandbox" for "b045617d-721d-477f-8db4-62c4fdc0c358" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"web-nginx-5f769fdc6-tt8mf_default\" network: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused\""" Jun 18 10:09:35 cloud kubelet[406675]: E0618 10:09:35.234612 406675 pod_workers.go:190] "Error syncing pod, skipping" err="failed to "KillPodSandbox" for "b045617d-721d-477f-8db4-62c4fdc0c358" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"web-nginx-5f769fdc6-tt8mf_default\" network: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/antrea/cni.sock: connect: connection refused\""" pod="default/web-nginx-5f769fdc6-tt8mf" podUID=b045617d-721d-477f-8db4-62c4fdc0c358 ^C root@cloud:~#
root@ubuntu:~# kubectl get pods NAME READY STATUS RESTARTS AGE debian-6c44fc6956-ltsrt 0/1 CrashLoopBackOff 5065 17d mc1 2/2 Running 0 17d my-deployment-68bdbbb5cc-bbszv 0/1 ImagePullBackOff 0 36d my-deployment-68bdbbb5cc-nrst9 0/1 ImagePullBackOff 0 36d my-deployment-68bdbbb5cc-rlgzt 0/1 ImagePullBackOff 0 36d my-nginx-5dc4865748-jqx54 1/1 Running 2 36d my-nginx-5dc4865748-pcrbg 1/1 Running 2 36d nginx 0/1 ImagePullBackOff 0 36d nginx-deployment-6b474476c4-r6z5b 1/1 Running 0 9d nginx-deployment-6b474476c4-w6xh9 1/1 Running 0 9d web-nginx-5f769fdc6-dlnqq 0/1 ContainerCreating 0 7m49s web-nginx-5f769fdc6-tt8mf 0/1 ContainerCreating 0 7m49s root@ubuntu:~# kubectl describe daemonset cloud -n kube-system | grep Image | cut -d "/" -f 2 Error from server (NotFound): daemonsets.apps "cloud" not found root@ubuntu:~# kubectl describe daemonset web-nginx-5f769fdc6-dlnqq -n default | grep Image | cut -d "/" -f 2 Error from server (NotFound): daemonsets.apps "web-nginx-5f769fdc6-dlnqq" not found root@ubuntu:~# root@ubuntu:~# kubectl delete --namespace=default deployment web-nginx deployment.apps "web-nginx" deleted root@ubuntu:~# kubectl get pods NAME READY STATUS RESTARTS AGE debian-6c44fc6956-ltsrt 0/1 CrashLoopBackOff 5067 17d mc1 2/2 Running 0 17d my-deployment-68bdbbb5cc-bbszv 0/1 ImagePullBackOff 0 36d my-deployment-68bdbbb5cc-nrst9 0/1 ImagePullBackOff 0 36d my-deployment-68bdbbb5cc-rlgzt 0/1 ImagePullBackOff 0 36d my-nginx-5dc4865748-jqx54 1/1 Running 2 36d my-nginx-5dc4865748-pcrbg 1/1 Running 2 36d nginx 0/1 ImagePullBackOff 0 36d nginx-deployment-6b474476c4-r6z5b 1/1 Running 0 9d nginx-deployment-6b474476c4-w6xh9 1/1 Running 0 9d web-nginx-5f769fdc6-dlnqq 0/1 Terminating 0 14m web-nginx-5f769fdc6-tt8mf 0/1 Terminating 0 14m root@ubuntu:~#
把
/etc/cni/net.d/10-antrea.conflist 删除
root@cloud:~# ls /var/lib/cni/ cache root@cloud:~# ls /etc/cni/net.d/ 10-antrea.conflist 10-flannel.conflist root@cloud:~# rm -rf /etc/cni/net.d/10-antrea.conflist root@cloud:~#
kubectl apply -f web-anti-affinity.yaml 成功了
root@ubuntu:~# kubectl get pods NAME READY STATUS RESTARTS AGE debian-6c44fc6956-ltsrt 0/1 Completed 5069 18d mc1 2/2 Running 0 17d my-deployment-68bdbbb5cc-bbszv 0/1 ImagePullBackOff 0 36d my-deployment-68bdbbb5cc-nrst9 0/1 ImagePullBackOff 0 36d my-deployment-68bdbbb5cc-rlgzt 0/1 ImagePullBackOff 0 36d my-nginx-5dc4865748-jqx54 1/1 Running 2 36d my-nginx-5dc4865748-pcrbg 1/1 Running 2 36d nginx 0/1 ImagePullBackOff 0 36d nginx-deployment-6b474476c4-r6z5b 1/1 Running 0 9d nginx-deployment-6b474476c4-w6xh9 1/1 Running 0 9d web-nginx-5f769fdc6-779lg 1/1 Running 0 2s web-nginx-5f769fdc6-sstj4 1/1 Running 0 2s root@ubuntu:~#
root@ubuntu:~# kubectl get pods web-nginx-5f769fdc6-779lg -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-nginx-5f769fdc6-779lg 1/1 Running 0 102s 10.244.2.4 cloud <none> <none> root@ubuntu:~# kubectl get pods web-nginx-5f769fdc6-sstj4 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-nginx-5f769fdc6-sstj4 1/1 Running 0 111s 10.244.2.5 cloud <none> <none> root@ubuntu:~#
root@ubuntu:~# kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS cloud Ready worker 15h v1.21.1 beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/arch=arm64,kubernetes.io/hostname=cloud,kubernetes.io/os=linux,node-role.kubernetes.io/worker=worker ubuntu Ready master 244d v1.18.1 beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/arch=arm64,kubernetes.io/hostname=ubuntu,kubernetes.io/os=linux,node-role.kubernetes.io/master= root@ubuntu:~#
两个pod没有打散哦
root@ubuntu:~# cat web-anti-affinity.yaml apiVersion: apps/v1 kind: Deployment metadata: name: web-nginx spec: selector: matchLabels: app: web-nginx replicas: 2 template: metadata: labels: app: web-nginx spec: affinity: #pod 反亲和性, 打散 web-nginx 各个副本 podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - web-nginx topologyKey: "kubernetes.io/hostname" containers: - image: nginx imagePullPolicy: IfNotPresent name: web2-worker ports: - containerPort: 8087 protocol: TCP
root@ubuntu:~# kubectl apply -f web-anti-affinity.yaml deployment.apps/web-nginx created root@ubuntu:~# kubectl get pods NAME READY STATUS RESTARTS AGE debian-6c44fc6956-ltsrt 0/1 CrashLoopBackOff 5071 18d mc1 2/2 Running 0 17d my-deployment-68bdbbb5cc-bbszv 0/1 ImagePullBackOff 0 36d my-deployment-68bdbbb5cc-nrst9 0/1 ImagePullBackOff 0 36d my-deployment-68bdbbb5cc-rlgzt 0/1 ImagePullBackOff 0 36d my-nginx-5dc4865748-jqx54 1/1 Running 2 36d my-nginx-5dc4865748-pcrbg 1/1 Running 2 36d nginx 0/1 ImagePullBackOff 0 36d nginx-deployment-6b474476c4-r6z5b 1/1 Running 0 9d nginx-deployment-6b474476c4-w6xh9 1/1 Running 0 9d web-nginx-7bdc6b976b-br45g 1/1 Running 0 9s web-nginx-7bdc6b976b-p9rxc 1/1 Running 0 9s root@ubuntu:~# kubectl get pods web-nginx-7bdc6b976b-br45g -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-nginx-7bdc6b976b-br45g 1/1 Running 0 23s 10.244.0.22 ubuntu <none> <none> root@ubuntu:~# kubectl get pods web-nginx-7bdc6b976b-p9rxc -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-nginx-7bdc6b976b-p9rxc 1/1 Running 0 34s 10.244.2.6 cloud <none> <none> root@ubuntu:~# cat web-anti-affinity.yaml
root@ubuntu:~# crictl inspect ca1b5c5a7aa2905d75a1f680ec774b09298ac09f03799b083e5eabffe0b5124a | grep -i pid "pid": 30603, "pid": 1 "type": "pid" root@ubuntu:~# nsenter -n --target 30603 root@ubuntu:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if673: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether ee:ee:d4:3a:73:67 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.0.22/24 brd 10.244.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::ecee:d4ff:fe3a:7367/64 scope link valid_lft forever preferred_lft forever root@ubuntu:~# ping 10.244.2.6 ----------访问不了 PING 10.244.2.6 (10.244.2.6) 56(84) bytes of data. From 10.244.2.0 icmp_seq=1 Destination Host Unreachable From 10.244.2.0 icmp_seq=2 Destination Host Unreachable From 10.244.2.0 icmp_seq=3 Destination Host Unreachable From 10.244.2.0 icmp_seq=4 Destination Host Unreachable From 10.244.2.0 icmp_seq=5 Destination Host Unreachable From 10.244.2.0 icmp_seq=6 Destination Host Unreachable From 10.244.2.0 icmp_seq=7 Destination Host Unreachable ^C --- 10.244.2.6 ping statistics --- 8 packets transmitted, 0 received, +7 errors, 100% packet loss, time 7112ms pipe 4 root@ubuntu:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.244.0.1 0.0.0.0 UG 0 0 0 eth0 10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 10.244.0.0 10.244.0.1 255.255.0.0 UG 0 0 0 eth0 root@ubuntu:~#
原来是antrea-gw0没有卸载干净，
root@ubuntu:~# ip a | grep 10.244.0.1 inet 10.244.0.1/24 brd 10.244.0.255 scope global antrea-gw0 inet 10.244.0.1/24 brd 10.244.0.255 scope global cni0 root@ubuntu:~# ip l del antrea-gw0 RTNETLINK answers: Operation not supported root@ubuntu:~# ip link del dev antrea-gw0 RTNETLINK answers: Operation not supported root@ubuntu:~# ip link delete antrea-gw0 RTNETLINK answers: Operation not supported root@ubuntu:~# ip a sh antrea-gw0 658: antrea-gw0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether f2:c9:55:a9:35:ad brd ff:ff:ff:ff:ff:ff inet 10.244.0.1/24 brd 10.244.0.255 scope global antrea-gw0 valid_lft forever preferred_lft forever inet6 fe80::f0c9:55ff:fea9:35ad/64 scope link valid_lft forever preferred_lft forever root@ubuntu:~# ip a flush antrea-gw0 root@ubuntu:~# ip a | grep 10.244.0.1 inet 10.244.0.1/24 brd 10.244.0.255 scope global cni0
root@ubuntu:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.244.0.1 0.0.0.0 UG 0 0 0 eth0 10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 10.244.0.0 10.244.0.1 255.255.0.0 UG 0 0 0 eth0 root@ubuntu:~# ping 10.244.2.6 PING 10.244.2.6 (10.244.2.6) 56(84) bytes of data. 64 bytes from 10.244.2.6: icmp_seq=1 ttl=62 time=0.442 ms 64 bytes from 10.244.2.6: icmp_seq=2 ttl=62 time=0.327 ms ^C --- 10.244.2.6 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1004ms rtt min/avg/max/mdev = 0.327/0.384/0.442/0.060 ms root@ubuntu:~#
root@ubuntu:~# kubectl exec -it web-nginx-7bdc6b976b-p9rxc -- /bin/bash root@web-nginx-7bdc6b976b-p9rxc:/# ip a bash: ip: command not found root@web-nginx-7bdc6b976b-p9rxc:/#
root@ubuntu:~# brctl show bridge name bridge id STP enabled interfaces cni0 8000.beca862286b8 no veth0dff33d9 veth224c8103 veth29d9bae9 veth38f93c57 veth3e31adfe veth45f94c26 veth7c984be7 veth8c55c45e veth8e1ca39c docker0 8000.02427319673d no vethc2ba676 vethf6368c9 virbr1 8000.cedad4f6fb17 no ftap0 root@ubuntu:~# ip a | grep 673 inet6 fe80::42:73ff:fe19:673d/64 scope link 673: veth8e1ca39c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default root@ubuntu:~#
root@ubuntu:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.10.34.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i3 10.10.100.0 0.0.0.0 255.255.255.0 U 0 0 0 peerh 10.10.104.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr1 10.10.104.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i2 10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0 10.244.2.0 10.244.2.0 255.255.255.0 UG 0 0 0 flannel.1 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 root@ubuntu:~#
由于flannel.1是一个vtep二层设备，所以需要根据vxlan的协议标准进行二层封装转发
mtu是1450
root@ubuntu:~# ip a show flannel.1 198: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether 3a:2b:ed:85:2f:74 brd ff:ff:ff:ff:ff:ff inet 10.244.0.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::382b:edff:fe85:2f74/64 scope link valid_lft forever preferred_lft forever root@ubuntu:~# tcpdump -i flannel.1 icmp -nv tcpdump: listening on flannel.1, link-type EN10MB (Ethernet), capture size 262144 bytes 11:02:45.407622 IP (tos 0x0, ttl 63, id 23793, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.0.22 > 10.244.2.6: ICMP echo request, id 41718, seq 1, length 64 11:02:45.408285 IP (tos 0x0, ttl 63, id 8634, offset 0, flags [none], proto ICMP (1), length 84) 10.244.2.6 > 10.244.0.22: ICMP echo reply, id 41718, seq 1, length 64 11:02:46.426314 IP (tos 0x0, ttl 63, id 23814, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.0.22 > 10.244.2.6: ICMP echo request, id 41718, seq 2, length 64 11:02:46.426436 IP (tos 0x0, ttl 63, id 8844, offset 0, flags [none], proto ICMP (1), length 84) 10.244.2.6 > 10.244.0.22: ICMP echo reply, id 41718, seq 2, length 64 ^C
root@ubuntu:~# bridge fdb show | grep flannel 72:d3:9a:47:fd:43 dev flannel.1 dst 10.10.16.47 self permanent root@ubuntu:~#
woker节点上
72:d3:9a:47:fd:43是flannel.1的mac
root@cloud:~# ip a sh flannel.1 14: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether 72:d3:9a:47:fd:43 brd ff:ff:ff:ff:ff:ff inet 10.244.2.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::70d3:9aff:fe47:fd43/64 scope link valid_lft forever preferred_lft forever root@cloud:~#
root@ubuntu:~# ping 10.244.2.6 PING 10.244.2.6 (10.244.2.6) 56(84) bytes of data. 64 bytes from 10.244.2.6: icmp_seq=1 ttl=62 time=0.425 ms ^C --- 10.244.2.6 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.425/0.425/0.425/0.000 ms root@ubuntu:~# ip n 10.244.0.1 dev eth0 lladdr be:ca:86:22:86:b8 DELAY root@ubuntu:~#
3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43
3a:2b:ed:85:2f:74是master flannel
root@ubuntu:~# ip link show flannel.1 198: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default link/ether 3a:2b:ed:85:2f:74 brd ff:ff:ff:ff:ff:ff
root@ubuntu:~# tcpdump -i enahisic2i0 host 10.10.16.47 and udp -eennv tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:15:48.634354 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 148: (tos 0x0, ttl 64, id 36729, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.82.47009 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 27945, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.0.22 > 10.244.2.6: ICMP echo request, id 47832, seq 12, length 64 11:15:48.634440 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 148: (tos 0x0, ttl 64, id 53685, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.47.55810 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 51026, offset 0, flags [none], proto ICMP (1), length 84) 10.244.2.6 > 10.244.0.22: ICMP echo reply, id 47832, seq 12, length 64 11:15:49.658347 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 148: (tos 0x0, ttl 64, id 36859, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.82.47009 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 27961, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.0.22 > 10.244.2.6: ICMP echo request, id 47832, seq 13, length 64 11:15:49.658438 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 148: (tos 0x0, ttl 64, id 53714, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.47.55810 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 51066, offset 0, flags [none], proto ICMP (1), length 84) 10.244.2.6 > 10.244.0.22: ICMP echo reply, id 47832, seq 13, length 64 ^C 4 packets captured 4 packets received by filter 0 packets dropped by kernel
worker节点上
root@cloud:~# kubectl get pods NAME READY STATUS RESTARTS AGE debian-6c44fc6956-ltsrt 0/1 CrashLoopBackOff 5077 18d mc1 2/2 Running 0 18d my-deployment-68bdbbb5cc-bbszv 0/1 ImagePullBackOff 0 36d my-deployment-68bdbbb5cc-nrst9 0/1 ImagePullBackOff 0 36d my-deployment-68bdbbb5cc-rlgzt 0/1 ImagePullBackOff 0 36d my-nginx-5dc4865748-jqx54 1/1 Running 2 36d my-nginx-5dc4865748-pcrbg 1/1 Running 2 36d nginx 0/1 ImagePullBackOff 0 36d nginx-deployment-6b474476c4-r6z5b 1/1 Running 0 9d nginx-deployment-6b474476c4-w6xh9 1/1 Running 0 9d web-nginx-7bdc6b976b-br45g 1/1 Running 0 29m web-nginx-7bdc6b976b-p9rxc 1/1 Running 0 29m root@cloud:~# kubectl -n default describe pod web-nginx-7bdc6b976b-p9rxc | grep Container Containers: Container ID: docker://eb20022b723803ef0cac93ba07c7584751821e388abc482709336777e7ba7c0a ContainersReady True Normal Pulled 29m kubelet, cloud Container image "nginx" already present on machine root@cloud:~# crictl inspect eb20022b723803ef0cac93ba07c7584751821e388abc482709336777e7ba7c0a | grep -i pid root@cloud:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES eb20022b7238 d0f910f78b97 "/docker-entrypoint.…" 30 minutes ago Up 30 minutes k8s_web2-worker_web-nginx-7bdc6b976b-p9rxc_default_30a12f33-be07-4294-bc43-88c6adb3ea18_0 01ddc519e9aa k8s.gcr.io/pause:3.2 "/pause" 30 minutes ago Up 30 minutes k8s_POD_web-nginx-7bdc6b976b-p9rxc_default_30a12f33-be07-4294-bc43-88c6adb3ea18_0 1812049e5eb5 7cf4a417daaa "/opt/bin/flanneld -…" About an hour ago Up About an hour k8s_kube-flannel_kube-flannel-ds-arm64-28rkj_kube-system_d683b27b-a6e8-448b-870b-709f07149187_0 9f396b91c6ea k8s.gcr.io/pause:3.2 "/pause" About an hour ago Up About an hour k8s_POD_kube-flannel-ds-arm64-28rkj_kube-system_d683b27b-a6e8-448b-870b-709f07149187_0 813710f5eac2 f782b1121865 "/usr/local/bin/kube…" 16 hours ago Up 16 hours k8s_kube-proxy_kube-proxy-nh2cp_kube-system_20b8a4ec-96e5-419f-8b6e-ff6137017318_0 596b821e1709 k8s.gcr.io/pause:3.2 "/pause" 16 hours ago Up 16 hours k8s_POD_kube-proxy-nh2cp_kube-system_20b8a4ec-96e5-419f-8b6e-ff6137017318_0 d8d153f65ace alpine "/bin/sh" 5 weeks ago Up 5 weeks alpine root@cloud:~# docker inspect eb20022b7238 | grep -i pid "Pid": 126118, "PidMode": "", "PidsLimit": null, root@cloud:~#
root@cloud:~# docker inspect eb20022b7238 | grep -i pid "Pid": 126118, "PidMode": "", "PidsLimit": null, root@cloud:~# nsenter -n --target 126118 root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 82:ea:86:37:c3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.2.6/24 brd 10.244.2.255 scope global eth0 valid_lft forever preferred_lft forever root@cloud:~#
root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 82:ea:86:37:c3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.2.6/24 brd 10.244.2.255 scope global eth0 valid_lft forever preferred_lft forever root@cloud:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.244.2.1 0.0.0.0 UG 0 0 0 eth0 10.244.0.0 10.244.2.1 255.255.0.0 UG 0 0 0 eth0 10.244.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 root@cloud:~#
root@cloud:~# brctl show bridge name bridge id STP enabled interfaces cni0 8000.0af9a27f2f2a no veth9cd09543 docker0 8000.0242c28bb536 no veth3c8f176 root@cloud:~# ip a | grep 20 7: dm-493626720dc1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 20: veth9cd09543@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default root@cloud:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 9.251.0.0 172.17.0.1 255.255.0.0 UG 0 0 0 docker0 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.99.1.231 10.10.16.82 255.255.255.255 UGH 0 0 0 enahisic2i0 10.110.79.116 10.10.16.82 255.255.255.255 UGH 0 0 0 enahisic2i0 10.110.171.213 10.10.16.82 255.255.255.255 UGH 0 0 0 enahisic2i0 10.244.0.0 10.244.0.0 255.255.255.0 UG 0 0 0 flannel.1 10.244.2.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 root@cloud:~# ip n 10.10.16.252 dev enahisic2i0 lladdr 00:23:81:26:93:6e STALE 10.10.16.81 dev enahisic2i0 lladdr 48:57:02:64:ea:1b STALE 10.244.2.6 dev cni0 lladdr 82:ea:86:37:c3:8d STALE 10.244.0.0 dev flannel.1 lladdr 3a:2b:ed:85:2f:74 PERMANENT 10.10.16.82 dev enahisic2i0 lladdr 48:57:02:64:e7:ab REACHABLE 10.10.16.253 dev enahisic2i0 lladdr 00:23:81:26:94:a0 STALE 10.10.16.254 dev enahisic2i0 lladdr f4:1d:6b:87:53:2a REACHABLE 10.10.16.250 dev enahisic2i0 lladdr 48:57:02:64:e7:ab STALE root@cloud:~# ip a sh flannel.1 14: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether 72:d3:9a:47:fd:43 brd ff:ff:ff:ff:ff:ff inet 10.244.2.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::70d3:9aff:fe47:fd43/64 scope link valid_lft forever preferred_lft forever root@cloud:~# ip a sh cni0 15: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether 0a:f9:a2:7f:2f:2a brd ff:ff:ff:ff:ff:ff inet 10.244.2.1/24 brd 10.244.2.255 scope global cni0 valid_lft forever preferred_lft forever inet6 fe80::8f9:a2ff:fe7f:2f2a/64 scope link valid_lft forever preferred_lft forever root@cloud:~#
root@cloud:~# bridge fdb show | grep flannel.1 3a:2b:ed:85:2f:74 dev flannel.1 dst 10.10.16.82 self permanent root@cloud:~#
master节点看不到worker节点上的容器
root@ubuntu:~# crictl inspect eb20022b723803ef0cac93ba07c7584751821e388abc482709336777e7ba7c0a | grep -i pid FATA[0000] Getting the status of the container "eb20022b723803ef0cac93ba07c7584751821e388abc482709336777e7ba7c0a" failed: rpc error: code = Unknown desc = an error occurred when try to find container "eb20022b723803ef0cac93ba07c7584751821e388abc482709336777e7ba7c0a": does not exist root@ubuntu:~#
root@cloud:~# bridge fdb show | grep 3a:2b:ed:85:2f:74 3a:2b:ed:85:2f:74 dev flannel.1 dst 10.10.16.82 self permanent root@cloud:~#
vxlan 封装
root@ubuntu:~# tcpdump -i enahisic2i0 host 10.10.16.47 and udp -nv tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:06:43.999084 IP (tos 0x0, ttl 64, id 45401, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.82.47009 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 IP (tos 0x0, ttl 63, id 56784, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.0.22 > 10.244.2.6: ICMP echo request, id 43612, seq 1, length 64 11:06:43.999600 IP (tos 0x0, ttl 64, id 23676, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.47.55810 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 IP (tos 0x0, ttl 63, id 45400, offset 0, flags [none], proto ICMP (1), length 84) 10.244.2.6 > 10.244.0.22: ICMP echo reply, id 43612, seq 1, length 64 11:06:45.018332 IP (tos 0x0, ttl 64, id 45473, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.82.47009 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 IP (tos 0x0, ttl 63, id 56853, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.0.22 > 10.244.2.6: ICMP echo request, id 43612, seq 2, length 64 11:06:45.018876 IP (tos 0x0, ttl 64, id 23723, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.47.55810 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 IP (tos 0x0, ttl 63, id 45528, offset 0, flags [none], proto ICMP (1), length 84) 10.244.2.6 > 10.244.0.22: ICMP echo reply, id 43612, seq 2, length 64
内层报文mac是两个flannel.1设备的mac
root@ubuntu:~# tcpdump -i enahisic2i0 host 10.10.16.47 and udp -eennv tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:15:48.634354 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 148: (tos 0x0, ttl 64, id 36729, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.82.47009 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 27945, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.0.22 > 10.244.2.6: ICMP echo request, id 47832, seq 12, length 64 11:15:48.634440 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 148: (tos 0x0, ttl 64, id 53685, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.47.55810 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 51026, offset 0, flags [none], proto ICMP (1), length 84) 10.244.2.6 > 10.244.0.22: ICMP echo reply, id 47832, seq 12, length 64 11:15:49.658347 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 148: (tos 0x0, ttl 64, id 36859, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.82.47009 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 27961, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.0.22 > 10.244.2.6: ICMP echo request, id 47832, seq 13, length 64 11:15:49.658438 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 148: (tos 0x0, ttl 64, id 53714, offset 0, flags [none], proto UDP (17), length 134) 10.10.16.47.55810 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 51066, offset 0, flags [none], proto ICMP (1), length 84) 10.244.2.6 > 10.244.0.22: ICMP echo reply, id 47832, seq 13, length 64
flannel 设备
root@ubuntu:~# ip -d link show flannel.1 198: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default link/ether 3a:2b:ed:85:2f:74 brd ff:ff:ff:ff:ff:ff promiscuity 0 vxlan id 1 local 10.10.16.82 dev enahisic2i0 srcport 0 0 dstport 8472 nolearning ttl inherit ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 28672 gso_max_segs 65535 root@ubuntu:~#
root@ubuntu:~# bridge fdb show dev flannel.1 72:d3:9a:47:fd:43 dst 10.10.16.47 self permanent
10.244.2.0/24的下一跳是 10.244.2.0（worker节点flannel.1的ip）
root@ubuntu:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.10.34.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i3 10.10.100.0 0.0.0.0 255.255.255.0 U 0 0 0 peerh 10.10.104.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr1 10.10.104.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i2 10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0 10.244.2.0 10.244.2.0 255.255.255.0 UG 0 0 0 flannel.1 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
10.244.2.0的mac是72:d3:9a:47:fd:43
root@ubuntu:~# ip n 10.244.0.2 dev cni0 lladdr 22:44:c3:88:d4:a4 REACHABLE 10.244.2.0 dev flannel.1 lladdr 72:d3:9a:47:fd:43 PERMANENT 10.10.16.81 dev enahisic2i0 lladdr 48:57:02:64:ea:1b STALE 10.10.100.82 dev peerh lladdr 1a:46:0b:ca:bc:7b STALE 10.10.16.254 dev enahisic2i0 lladdr f4:1d:6b:87:53:2a REACHABLE 10.10.16.47 dev enahisic2i0 lladdr 9c:52:f8:67:c4:d3 REACHABLE 10.244.0.20 dev cni0 lladdr 6e:5a:30:bc:6d:5b STALE 10.10.34.251 dev enahisic2i3 FAILED 10.10.16.250 dev enahisic2i0 lladdr 48:57:02:64:ea:1b STALE 10.244.0.3 dev cni0 lladdr b6:3f:e6:3b:a0:cc REACHABLE 10.10.16.27 dev enahisic2i0 lladdr 9c:52:f8:67:c6:47 STALE 10.244.0.4 dev cni0 lladdr 22:9a:d3:1f:e3:49 STALE 10.244.0.22 dev cni0 lladdr ee:ee:d4:3a:73:67 STALE 172.17.0.4 dev docker0 lladdr 02:42:ac:11:00:04 STALE 172.17.0.3 dev docker0 lladdr 02:42:ac:11:00:03 STALE 10.10.16.48 dev enahisic2i0 FAILED 10.244.0.19 dev cni0 lladdr fa:66:b3:ab:05:9f STALE 10.244.0.5 dev cni0 lladdr 32:26:5a:e7:0d:83 STALE 10.10.16.1 dev enahisic2i0 lladdr 48:57:02:64:ee:9b STALE root@ubuntu:~#
72:d3:9a:47:fd:43的remote vtep ip 是10.10.16.47
root@ubuntu:~# bridge fdb show dev flannel.1 72:d3:9a:47:fd:43 dst 10.10.16.47 self permanent
flannel配置
root@ubuntu:~# cat /run/flannel/subnet.env FLANNEL_NETWORK=10.244.0.0/16 FLANNEL_SUBNET=10.244.0.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=true root@ubuntu:~#
root@cloud:~# cat /run/flannel/subnet.env FLANNEL_NETWORK=10.244.0.0/16 FLANNEL_SUBNET=10.244.2.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=true root@cloud:~#
node port
root@cloud:~# telnet 10.244.0.22 8087 ------无法访问 Trying 10.244.0.22... telnet: Unable to connect to remote host: Connection refused root@cloud:~# telnet 10.244.0.22 80 Trying 10.244.0.22... Connected to 10.244.0.22. Escape character is '^]'. ^CConnection closed by foreign host. root@cloud:~#
root@ubuntu:~# kubectl apply -f web-ngx-svc.yml service/nodeport-svc created root@ubuntu:~# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 244d my-nginx ClusterIP 10.110.79.116 <none> 8280/TCP 36d my-nginx-np NodePort 10.99.1.231 <none> 8081:31199/TCP 36d nodeport-svc NodePort 10.97.11.232 <none> 3000:30090/TCP 5s web2 NodePort 10.110.171.213 <none> 8097:31866/TCP 20d root@ubuntu:~# cat web-ngx-svc.yml apiVersion: v1 kind: Service metadata: name: nodeport-svc spec: type: NodePort selector: app: web-nginx ports: - protocol: TCP port: 3000 targetPort: 8087 nodePort: 30090 root@ubuntu:~#
-
nodePort 是节点上监听的端口
-
port 是 ClusterIP 上监听的端口
-
targetPort 是 Pod 监听的端口
port
port是k8s集群内部访问service的端口，即通过clusterIP: port可以访问到某个service
nodePort
nodePort是外部访问k8s集群中service的端口，通过nodeIP: nodePort可以从外部访问到某个service。
targetPort
targetPort是pod的端口，从port和nodePort来的流量经过kube-proxy流入到后端pod的targetPort上，最后进入容器。
containerPort
containerPort是pod内部容器的端口，targetPort映射到containerPort。
图解
nodeport-svc NodePort 10.97.11.232 <none> 3000:30090/TCP 5m25s
从10.10.16.81上访问，无法访问
[root@bogon ~]# telnet 10.10.16.82 3000 Trying 10.10.16.82... telnet: connect to address 10.10.16.82: Connection refused [root@bogon ~]#
[root@bogon ~]# telnet 10.10.16.82 30090 Trying 10.10.16.82...
containerPort有问题
root@ubuntu:~# cat web-anti-affinity.yaml apiVersion: apps/v1 kind: Deployment metadata: name: web-nginx spec: selector: matchLabels: app: web-nginx replicas: 2 template: metadata: labels: app: web-nginx spec: affinity: #pod 反亲和性, 打散 web-nginx 各个副本 podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - web-nginx topologyKey: "kubernetes.io/hostname" containers: - image: nginx imagePullPolicy: IfNotPresent name: web2-worker ports: - containerPort: 8087 protocol: TCP
更改
kubectl edit deployment web-nginx
- containerPort: 80
编辑
kubectl edit svc/nodeport-svc
更改nodeport-svc
[root@bogon ~]# telnet 10.10.16.82 30090 Trying 10.10.16.82... Connected to 10.10.16.82. Escape character is '^]'. ^CConnection closed by foreign host. You have new mail in /var/spool/mail/root [root@bogon ~]#
node port 上抓包
root@ubuntu:~# tcpdump -i enahisic2i0 tcp and port 30090 -ennvv tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes 14:51:19.081510 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 60738, offset 0, flags [DF], proto TCP (6), length 60) 10.10.16.81.59402 > 10.10.16.82.30090: Flags [S], cksum 0xc918 (correct), seq 2066571357, win 29200, options [mss 1460,sackOK,TS val 16096779 ecr 0,nop,wscale 7], length 0 14:51:19.082383 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.10.16.82.30090 > 10.10.16.81.59402: Flags [S.], cksum 0x18f4 (correct), seq 2393608867, ack 2066571358, win 64308, options [mss 1410,sackOK,TS val 2683332065 ecr 16096779,nop,wscale 7], length 0 14:51:19.082441 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 60739, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.81.59402 > 10.10.16.82.30090: Flags [.], cksum 0x41dd (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 16096780 ecr 2683332065], length 0 14:51:21.830731 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 71: (tos 0x10, ttl 64, id 60740, offset 0, flags [DF], proto TCP (6), length 57) 10.10.16.81.59402 > 10.10.16.82.30090: Flags [P.], cksum 0x3121 (correct), seq 1:6, ack 1, win 229, options [nop,nop,TS val 16099528 ecr 2683332065], length 5 14:51:21.831085 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 62, id 31192, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.30090 > 10.10.16.81.59402: Flags [.], cksum 0x2b4d (correct), seq 1, ack 6, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 0 14:51:21.831218 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 375: (tos 0x0, ttl 62, id 31193, offset 0, flags [DF], proto TCP (6), length 361) 10.10.16.82.30090 > 10.10.16.81.59402: Flags [P.], cksum 0x81f9 (correct), seq 1:310, ack 6, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 309 14:51:21.831260 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 62, id 31194, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.30090 > 10.10.16.81.59402: Flags [F.], cksum 0x2a17 (correct), seq 310, ack 6, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 0 14:51:21.831270 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 60741, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.81.59402 > 10.10.16.82.30090: Flags [.], cksum 0x2b22 (correct), seq 6, ack 310, win 237, options [nop,nop,TS val 16099528 ecr 2683334814], length 0 14:51:21.831341 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 60742, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.81.59402 > 10.10.16.82.30090: Flags [F.], cksum 0x2b20 (correct), seq 6, ack 311, win 237, options [nop,nop,TS val 16099528 ecr 2683334814], length 0 14:51:21.831464 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 62, id 31195, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.30090 > 10.10.16.81.59402: Flags [.], cksum 0x2a16 (correct), seq 311, ack 7, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 0
worker 节点 pod ns tcpdump
root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 82:ea:86:37:c3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.2.6/24 brd 10.244.2.255 scope global eth0 valid_lft forever preferred_lft forever root@cloud:~# ip a | grep 72:d3:9a:47:fd:43 root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 82:ea:86:37:c3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.2.6/24 brd 10.244.2.255 scope global eth0 valid_lft forever preferred_lft forever root@cloud:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.244.2.1 0.0.0.0 UG 0 0 0 eth0 10.244.0.0 10.244.2.1 255.255.0.0 UG 0 0 0 eth0 10.244.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 root@cloud:~# telnet 10.244.0.22 8087 Trying 10.244.0.22... telnet: Unable to connect to remote host: Connection refused root@cloud:~# telnet 10.244.0.22 80 Trying 10.244.0.22... Connected to 10.244.0.22. Escape character is '^]'. ^CConnection closed by foreign host. root@cloud:~# telnet 10.244.0.22 8087 Trying 10.244.0.22... telnet: Unable to connect to remote host: Connection refused
root@cloud:~# tcpdump -i eth0 tcp and port 80 -ennvv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:51:19.079556 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 62, id 60738, offset 0, flags [DF], proto TCP (6), length 60)
10.244.0.0.51150 > 10.244.2.6.80: Flags [S], cksum 0x7b58 (correct), seq 2066571357, win 29200, options [mss 1460,sackOK,TS val 16096779 ecr 0,nop,wscale 7], length 0
14:51:19.079633 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.244.2.6.80 > 10.244.0.0.51150: Flags [S.], cksum 0x181c (incorrect -> 0xcb33), seq 2393608867, ack 2066571358, win 64308, options [mss 1410,sackOK,TS val 2683332065 ecr 16096779,nop,wscale 7], length 0
14:51:19.079874 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 62, id 60739, offset 0, flags [DF], proto TCP (6), length 52)
10.244.0.0.51150 > 10.244.2.6.80: Flags [.], cksum 0xf41c (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 16096780 ecr 2683332065], length 0
14:51:21.828291 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 71: (tos 0x10, ttl 62, id 60740, offset 0, flags [DF], proto TCP (6), length 57)
10.244.0.0.51150 > 10.244.2.6.80: Flags [P.], cksum 0xe360 (correct), seq 1:6, ack 1, win 229, options [nop,nop,TS val 16099528 ecr 2683332065], length 5: HTTP
14:51:21.828347 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 31192, offset 0, flags [DF], proto TCP (6), length 52)
10.244.2.6.80 > 10.244.0.0.51150: Flags [.], cksum 0x1814 (incorrect -> 0xdd8c), seq 1, ack 6, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 0
14:51:21.828441 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 375: (tos 0x0, ttl 64, id 31193, offset 0, flags [DF], proto TCP (6), length 361)
10.244.2.6.80 > 10.244.0.0.51150: Flags [P.], cksum 0x1949 (incorrect -> 0x3439), seq 1:310, ack 6, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 309: HTTP, length: 309
HTTP/1.1 400 Bad Request
Server: nginx/1.19.5
Date: Fri, 18 Jun 2021 06:51:21 GMT
Content-Type: text/html
Content-Length: 157
Connection: close
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.19.5</center>
</body>
</html>
14:51:21.828560 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 31194, offset 0, flags [DF], proto TCP (6), length 52)
10.244.2.6.80 > 10.244.0.0.51150: Flags [F.], cksum 0x1814 (incorrect -> 0xdc56), seq 310, ack 6, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 0
14:51:21.828678 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 62, id 60741, offset 0, flags [DF], proto TCP (6), length 52)
10.244.0.0.51150 > 10.244.2.6.80: Flags [.], cksum 0xdd61 (correct), seq 6, ack 310, win 237, options [nop,nop,TS val 16099528 ecr 2683334814], length 0
14:51:21.828747 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 62, id 60742, offset 0, flags [DF], proto TCP (6), length 52)
10.244.0.0.51150 > 10.244.2.6.80: Flags [F.], cksum 0xdd5f (correct), seq 6, ack 311, win 237, options [nop,nop,TS val 16099528 ecr 2683334814], length 0
14:51:21.828765 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 31195, offset 0, flags [DF], proto TCP (6), length 52)
10.244.2.6.80 > 10.244.0.0.51150: Flags [.], cksum 0x1814 (incorrect -> 0xdc55), seq 311, ack 7, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 0
10.244.0.0 是 master flannel.1的ip
root@ubuntu:~# ip a | grep 10.244.0.0 inet 10.244.0.0/32 scope global flannel.1 root@ubuntu:~#
root@ubuntu:~# ip a sh flannel.1 198: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether 3a:2b:ed:85:2f:74 brd ff:ff:ff:ff:ff:ff inet 10.244.0.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::382b:edff:fe85:2f74/64 scope link valid_lft forever preferred_lft forever root@ubuntu:~#
0a:f9:a2:7f:2f:2a 是cni0的mac
root@cloud:~# ip a | grep 0a:f9:a2:7f:2f:2a link/ether 0a:f9:a2:7f:2f:2a brd ff:ff:ff:ff:ff:ff root@cloud:~# ip a | grep 0a:f9:a2:7f:2f:2a -A 3 link/ether 0a:f9:a2:7f:2f:2a brd ff:ff:ff:ff:ff:ff inet 10.244.2.1/24 brd 10.244.2.255 scope global cni0 valid_lft forever preferred_lft forever inet6 fe80::8f9:a2ff:fe7f:2f2a/64 scope link root@cloud:~# ip a sh cni0 15: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether 0a:f9:a2:7f:2f:2a brd ff:ff:ff:ff:ff:ff inet 10.244.2.1/24 brd 10.244.2.255 scope global cni0 valid_lft forever preferred_lft forever inet6 fe80::8f9:a2ff:fe7f:2f2a/64 scope link valid_lft forever preferred_lft forever root@cloud:~#
82:ea:86:37:c3:8d
root@cloud:~# ip a | grep 82:ea:86:37:c3:8d link/ether 82:ea:86:37:c3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 82:ea:86:37:c3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.2.6/24 brd 10.244.2.255 scope global eth0 valid_lft forever preferred_lft forever root@cloud:~#
在cloud 节点上抓vxlan报文
root@cloud:~# tcpdump -i enahisic2i0 udp and host 10.10.16.82 -eennv tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes 14:51:19.079030 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 64, id 32281, offset 0, flags [none], proto UDP (17), length 110) 10.10.16.82.22431 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 63, id 60738, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.0.51150 > 10.244.2.6.80: Flags [S], cksum 0x7b58 (correct), seq 2066571357, win 29200, options [mss 1460,sackOK,TS val 16096779 ecr 0,nop,wscale 7], length 0 14:51:19.079680 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 64, id 9365, offset 0, flags [none], proto UDP (17), length 110) 10.10.16.47.48491 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
回复报文 10.244.2.6.80 > 10.244.0.0.51150: Flags [S.], cksum 0xcb33 (correct), seq 2393608867, ack 2066571358, win 64308, options [mss 1410,sackOK,TS val 2683332065 ecr 16096779,nop,wscale 7], length 0 14:51:19.079853 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 32282, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.82.22431 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 60739, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.51150 > 10.244.2.6.80: Flags [.], cksum 0xf41c (correct), ack 1, win 229, options [nop,nop,TS val 16096780 ecr 2683332065], length 0 14:51:21.828191 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 121: (tos 0x0, ttl 64, id 32342, offset 0, flags [none], proto UDP (17), length 107) 10.10.16.82.22431 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 71: (tos 0x10, ttl 63, id 60740, offset 0, flags [DF], proto TCP (6), length 57) 10.244.0.0.51150 > 10.244.2.6.80: Flags [P.], cksum 0xe360 (correct), seq 1:6, ack 1, win 229, options [nop,nop,TS val 16099528 ecr 2683332065], length 5: HTTP 14:51:21.828393 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 9965, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.47.48491 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 31192, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.51150: Flags [.], cksum 0xdd8c (correct), ack 6, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 0 14:51:21.828526 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 425: (tos 0x0, ttl 64, id 9966, offset 0, flags [none], proto UDP (17), length 411) 10.10.16.47.48491 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 375: (tos 0x0, ttl 63, id 31193, offset 0, flags [DF], proto TCP (6), length 361) 10.244.2.6.80 > 10.244.0.0.51150: Flags [P.], cksum 0x3439 (correct), seq 1:310, ack 6, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 309: HTTP, length: 309 HTTP/1.1 400 Bad Request Server: nginx/1.19.5 Date: Fri, 18 Jun 2021 06:51:21 GMT Content-Type: text/html Content-Length: 157 Connection: close <html> <head><title>400 Bad Request</title></head> <body> <center><h1>400 Bad Request</h1></center> <hr><center>nginx/1.19.5</center> </body> </html> 14:51:21.828576 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 9967, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.47.48491 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 31194, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.51150: Flags [F.], cksum 0xdc56 (correct), seq 310, ack 6, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 0 14:51:21.828659 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 32343, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.82.22431 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 60741, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.51150 > 10.244.2.6.80: Flags [.], cksum 0xdd61 (correct), ack 310, win 237, options [nop,nop,TS val 16099528 ecr 2683334814], length 0 14:51:21.828734 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 32344, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.82.22431 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 60742, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.51150 > 10.244.2.6.80: Flags [F.], cksum 0xdd5f (correct), seq 6, ack 311, win 237, options [nop,nop,TS val 16099528 ecr 2683334814], length 0 14:51:21.828793 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 9968, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.47.48491 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 31195, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.51150: Flags [.], cksum 0xdc55 (correct), ack 7, win 503, options [nop,nop,TS val 2683334814 ecr 16099528], length 0 ^C 10 packets captured 10 packets received by filter 0 packets dropped by kernel
转发到master上的pod
[root@bogon ~]# telnet 10.10.16.82 30090 Trying 10.10.16.82... Connected to 10.10.16.82. Escape character is '^]'. Connection closed by foreign host. You have mail in /var/spool/mail/root [root@bogon ~]
root@ubuntu:~# conntrack -L -o ktimestamp | grep 30090 tcp 6 86374 ESTABLISHED src=10.10.16.81 dst=10.10.16.82 sport=59600 dport=30090 src=10.244.0.22 dst=10.244.0.1 sport=80 dport=9351 [ASSURED] mark=0 use=1 conntrack v1.4.4 (conntrack-tools): 157 flow entries have been shown. root@ubuntu:~#
root@ubuntu:~# tcpdump -i eth0 tcp and port 80 -ennvv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:03:43.259291 be:ca:86:22:86:b8 > ee:ee:d4:3a:73:67, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 63, id 59038, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.1.9351 > 10.244.0.22.80: Flags [S], cksum 0x1226 (correct), seq 1169049935, win 29200, options [mss 1460,sackOK,TS val 16840950 ecr 0,nop,wscale 7], length 0 15:03:43.259337 ee:ee:d4:3a:73:67 > be:ca:86:22:86:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.22.80 > 10.244.0.1.9351: Flags [S.], cksum 0x162d (incorrect -> 0x40ca), seq 3025682805, ack 1169049936, win 64308, options [mss 1410,sackOK,TS val 3735122920 ecr 16840950,nop,wscale 7], length 0 15:03:43.259466 be:ca:86:22:86:b8 > ee:ee:d4:3a:73:67, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 59039, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.1.9351 > 10.244.0.22.80: Flags [.], cksum 0x69b4 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 16840950 ecr 3735122920], length 0 15:04:43.300895 ee:ee:d4:3a:73:67 > be:ca:86:22:86:b8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 58214, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.22.80 > 10.244.0.1.9351: Flags [F.], cksum 0x1625 (incorrect -> 0x7e17), seq 1, ack 1, win 503, options [nop,nop,TS val 3735182961 ecr 16840950], length 0 15:04:43.301177 be:ca:86:22:86:b8 > ee:ee:d4:3a:73:67, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 59040, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.1.9351 > 10.244.0.22.80: Flags [F.], cksum 0x949d (correct), seq 1, ack 2, win 229, options [nop,nop,TS val 16900992 ecr 3735182961], length 0 15:04:43.301212 ee:ee:d4:3a:73:67 > be:ca:86:22:86:b8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 58215, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.22.80 > 10.244.0.1.9351: Flags [.], cksum 0x1625 (incorrect -> 0x938a), seq 2, ack 2, win 503, options [nop,nop,TS val 3735182962 ecr 16900992], length 0
10.244.0.1是cni0接口的ip不是flannel
root@ubuntu:~# ip a sh cni0 193: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether be:ca:86:22:86:b8 brd ff:ff:ff:ff:ff:ff inet 10.244.0.1/24 brd 10.244.0.255 scope global cni0 valid_lft forever preferred_lft forever inet6 fe80::bcca:86ff:fe22:86b8/64 scope link valid_lft forever preferred_lft forever root@ubuntu:~#
worker节点上的contrack
root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 82:ea:86:37:c3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.2.6/24 brd 10.244.2.255 scope global eth0 valid_lft forever preferred_lft forever
root@cloud:~# tcpdump -i eth0 tcp and port 80 -ennvv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:10:11.390387 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 62, id 24198, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.0.17110 > 10.244.2.6.80: Flags [S], cksum 0x5918 (correct), seq 565778920, win 29200, options [mss 1460,sackOK,TS val 17229084 ecr 0,nop,wscale 7], length 0 15:10:11.390454 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.2.6.80 > 10.244.0.0.17110: Flags [S.], cksum 0x181c (incorrect -> 0x4fce), seq 2968027747, ack 565778921, win 64308, options [mss 1410,sackOK,TS val 2684464376 ecr 17229084,nop,wscale 7], length 0 15:10:11.390693 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 62, id 24199, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.17110 > 10.244.2.6.80: Flags [.], cksum 0x78b8 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 17229084 ecr 2684464376], length 0 15:11:11.450903 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 54256, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.17110: Flags [F.], cksum 0x1814 (incorrect -> 0x8d08), seq 1, ack 1, win 503, options [nop,nop,TS val 2684524436 ecr 17229084], length 0 15:11:11.451320 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 62, id 24200, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.17110 > 10.244.2.6.80: Flags [F.], cksum 0xa37b (correct), seq 1, ack 2, win 229, options [nop,nop,TS val 17289145 ecr 2684524436], length 0 15:11:11.451343 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 54257, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.17110: Flags [.], cksum 0x1814 (incorrect -> 0xa268), seq 2, ack 2, win 503, options [nop,nop,TS val 2684524437 ecr 17289145], length 0
root@cloud:~# conntrack -L -o ktimestamp | grep 17110 conntrack v1.4.4 (conntrack-tools): 9 flow entries have been shown. tcp 6 106 TIME_WAIT src=10.244.0.0 dst=10.244.2.6 sport=17110 dport=80 src=10.244.2.6 dst=10.244.0.0 sport=80 dport=17110 [ASSURED] mark=0 use=1 root@cloud:~# conntrack -L -o ktimestamp | grep 17110 conntrack v1.4.4 (conntrack-tools): 9 flow entries have been shown. tcp 6 103 TIME_WAIT src=10.244.0.0 dst=10.244.2.6 sport=17110 dport=80 src=10.244.2.6 dst=10.244.0.0 sport=80 dport=17110 [ASSURED] mark=0 use=1 root@cloud:~# ip a sh flannl.1 Device "flannl.1" does not exist. root@cloud:~# ip a sh flannel.1 14: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether 72:d3:9a:47:fd:43 brd ff:ff:ff:ff:ff:ff inet 10.244.2.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::70d3:9aff:fe47:fd43/64 scope link valid_lft forever preferred_lft forever root@cloud:~#
demo2
root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 82:ea:86:37:c3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.2.6/24 brd 10.244.2.255 scope global eth0 valid_lft forever preferred_lft forever
root@cloud:~# tcpdump -i eth0 tcp and port 80 -ennvv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:10:11.390387 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 62, id 24198, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.0.17110 > 10.244.2.6.80: Flags [S], cksum 0x5918 (correct), seq 565778920, win 29200, options [mss 1460,sackOK,TS val 17229084 ecr 0,nop,wscale 7], length 0 15:10:11.390454 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.2.6.80 > 10.244.0.0.17110: Flags [S.], cksum 0x181c (incorrect -> 0x4fce), seq 2968027747, ack 565778921, win 64308, options [mss 1410,sackOK,TS val 2684464376 ecr 17229084,nop,wscale 7], length 0 15:10:11.390693 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 62, id 24199, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.17110 > 10.244.2.6.80: Flags [.], cksum 0x78b8 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 17229084 ecr 2684464376], length 0 15:11:11.450903 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 54256, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.17110: Flags [F.], cksum 0x1814 (incorrect -> 0x8d08), seq 1, ack 1, win 503, options [nop,nop,TS val 2684524436 ecr 17229084], length 0 15:11:11.451320 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 62, id 24200, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.17110 > 10.244.2.6.80: Flags [F.], cksum 0xa37b (correct), seq 1, ack 2, win 229, options [nop,nop,TS val 17289145 ecr 2684524436], length 0 15:11:11.451343 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 54257, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.17110: Flags [.], cksum 0x1814 (incorrect -> 0xa268), seq 2, ack 2, win 503, options [nop,nop,TS val 2684524437 ecr 17289145], length 0 15:13:54.600850 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 62, id 9591, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.0.31890 > 10.244.2.6.80: Flags [S], cksum 0xd396 (correct), seq 1670495208, win 29200, options [mss 1460,sackOK,TS val 17452293 ecr 0,nop,wscale 7], length 0 15:13:54.600913 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.2.6.80 > 10.244.0.0.31890: Flags [S.], cksum 0x181c (incorrect -> 0x52f7), seq 1541400276, ack 1670495209, win 64308, options [mss 1410,sackOK,TS val 2684687586 ecr 17452293,nop,wscale 7], length 0 15:13:54.601144 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 62, id 9592, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.31890 > 10.244.2.6.80: Flags [.], cksum 0x7be1 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 17452293 ecr 2684687586], length 0
master节点上的contrack
root@ubuntu:~# conntrack -L -o ktimestamp | grep 31890 tcp 6 86376 ESTABLISHED src=10.10.16.81 dst=10.10.16.82 sport=59720 dport=30090 src=10.244.2.6 dst=10.244.0.0 sport=80 dport=31890 [ASSURED] mark=0 use=2 conntrack v1.4.4 (conntrack-tools): 158 flow entries have been shown. root@ubuntu:~#
worker节点上的contrack
conntrack v1.4.4 (conntrack-tools): 9 flow entries have been shown. tcp 6 86384 ESTABLISHED src=10.244.0.0 dst=10.244.2.6 sport=31890 dport=80 src=10.244.2.6 dst=10.244.0.0 sport=80 dport=31890 [ASSURED] mark=0 use=1 root@cloud:~# conntrack -L -o ktimestamp | grep 31890 conntrack v1.4.4 (conntrack-tools): 7 flow entries have been shown. tcp 6 9 TIME_WAIT src=10.244.0.0 dst=10.244.2.6 sport=31890 dport=80 src=10.244.2.6 dst=10.244.0.0 sport=80 dport=31890 [ASSURED] mark=0 use=1 root@cloud:~#
使用etcdctl访问kubernetes数据
Kubenretes1.6中使用etcd V3版本的API，使用etcdctl直接ls的话只能看到/kube-centos一个路径。需要在命令前加上ETCDCTL_API=3这个环境变量才能看到kuberentes在etcd中保存的数据。
root@ubuntu:~/etcd-v3.5.0-linux-arm64# ETCDCTL_API=3 etcdctl get /registry/namespaces/default -w=json|python -m json.tool Error: context deadline exceeded No JSON object could be decoded root@ubuntu:~/etcd-v3.5.0-linux-arm64#
如果是使用 kubeadm 创建的集群，在 Kubenretes 1.11 中，etcd 默认使用 tls ，这时你可以在 master 节点上使用以下命令来访问 etcd ：
root@ubuntu:~/etcd-v3.5.0-linux-arm64# ETCDCTL_API=3 ./etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt > --cert=/etc/kubernetes/pki/etcd/peer.crt > --key=/etc/kubernetes/pki/etcd/peer.key > get /registry/namespaces/default -w=json | jq . { "header": { "cluster_id": 755078206002876000, "member_id": 9167673865571135000, "revision": 46601099, "raft_term": 2 }, "kvs": [ { "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvZGVmYXVsdA==", "create_revision": 152, "mod_revision": 152, "version": 1, "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlErIBCpcBCgdkZWZhdWx0EgAaACIAKiQ5MTcxNWVkNy1hM2VjLTQ4MzAtOTk2ZS0zNTM1MmY2Y2Y2NDQyADgAQggIz8il/AUQAHoAigFPCg5rdWJlLWFwaXNlcnZlchIGVXBkYXRlGgJ2MSIICM/IpfwFEAAyCEZpZWxkc1YxOh0KG3siZjpzdGF0dXMiOnsiZjpwaGFzZSI6e319fRIMCgprdWJlcm5ldGVzGggKBkFjdGl2ZRoAIgA=" } ], "count": 1 } root@ubuntu:~/etcd-v3.5.0-linux-arm64# ETCDCTL_API=3 ./etcdctl get /registry/namespaces --prefix -w=json|python -m json.tool {"level":"warn","ts":"2021-06-18T11:56:38.960+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x40004aaa80/#initially=[127.0.0.1:2379]","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection closed"} Error: context deadline exceeded No JSON object could be decoded root@ubuntu:~/etcd-v3.5.0-linux-arm64#
iptables 和30090
worker 节点
root@cloud:~# iptables -nvL -t nat | grep 30090 0 0 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nodeport-svc: */ tcp dpt:30090 0 0 KUBE-SVC-GFPAJ7EGCNM4QF4H tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nodeport-svc: */ tcp dpt:30090 root@cloud:~#
master pod tcpdump
root@ubuntu:~# tcpdump -i eth0 tcp and port 80 -ennvv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:24:46.235872 be:ca:86:22:86:b8 > ee:ee:d4:3a:73:67, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 55, id 38770, offset 0, flags [DF], proto TCP (6), length 60) 10.244.2.0.19231 > 10.244.0.22.80: Flags [S], cksum 0xb1ea (correct), seq 799646685, win 64240, options [mss 1460,nop,wscale 8,sackOK,TS val 857682193 ecr 0], length 0 15:24:46.235924 ee:ee:d4:3a:73:67 > be:ca:86:22:86:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.22.80 > 10.244.2.0.19231: Flags [S.], cksum 0x182c (incorrect -> 0x362f), seq 4026376631, ack 799646686, win 64308, options [mss 1410,sackOK,TS val 741064120 ecr 857682193,nop,wscale 7], length 0 15:24:46.237318 be:ca:86:22:86:b8 > ee:ee:d4:3a:73:67, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 55, id 38773, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.0.19231 > 10.244.0.22.80: Flags [.], cksum 0x5dfb (correct), seq 1, ack 1, win 513, options [nop,nop,TS val 857682195 ecr 741064120], length 0 ^C 3 packets captured 3 packets received by filter 0 packets dropped by kernel root@ubuntu:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if673: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether ee:ee:d4:3a:73:67 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.0.22/24 brd 10.244.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::ecee:d4ff:fe3a:7367/64 scope link valid_lft forever preferred_lft forever root@ubuntu:~#
master tcp vxlan
访问http://10.10.16.47:30090/从worker封装vxlan转发到master 的 pod
tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:24:46.235778 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 64, id 50002, offset 0, flags [none], proto UDP (17), length 110) 10.10.16.47.11568 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 56, id 38770, offset 0, flags [DF], proto TCP (6), length 60) 10.244.2.0.19231 > 10.244.0.22.80: Flags [S], cksum 0xb1ea (correct), seq 799646685, win 64240, options [mss 1460,nop,wscale 8,sackOK,TS val 857682193 ecr 0], length 0 15:24:46.235969 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 64, id 1499, offset 0, flags [none], proto UDP (17), length 110) 10.10.16.82.58148 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.22.80 > 10.244.2.0.19231: Flags [S.], cksum 0x362f (correct), seq 4026376631, ack 799646686, win 64308, options [mss 1410,sackOK,TS val 741064120 ecr 857682193,nop,wscale 7], length 0 15:24:46.237290 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 50003, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.47.11568 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 56, id 38773, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.0.19231 > 10.244.0.22.80: Flags [.], cksum 0x5dfb (correct), ack 1, win 513, options [nop,nop,TS val 857682195 ecr 741064120], length 0 15:25:31.249957 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 105: (tos 0x0, ttl 64, id 54722, offset 0, flags [none], proto UDP (17), length 91) 10.10.16.47.59334 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 55: (tos 0x0, ttl 56, id 38782, offset 0, flags [DF], proto TCP (6), length 41) 10.244.2.0.19231 > 10.244.0.22.80: Flags [.], cksum 0xe528 (correct), seq 0:1, ack 1, win 513, length 1: HTTP 15:25:31.250099 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 128: (tos 0x0, ttl 64, id 7543, offset 0, flags [none], proto UDP (17), length 114) 10.10.16.82.58148 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 63, id 53960, offset 0, flags [DF], proto TCP (6), length 64) 10.244.0.22.80 > 10.244.2.0.19231: Flags [.], cksum 0xd108 (correct), ack 1, win 503, options [nop,nop,TS val 741109134 ecr 857682195,nop,nop,sack 1 {0:1}], length 0 15:25:46.246753 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 9669, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.82.58148 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 53961, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.22.80 > 10.244.2.0.19231: Flags [F.], cksum 0x7398 (correct), seq 1, ack 1, win 503, options [nop,nop,TS val 741124131 ecr 857682195], length 0 15:25:46.249499 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 57649, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.47.59334 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 56, id 38784, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.0.19231 > 10.244.0.22.80: Flags [.], cksum 0x8921 (correct), ack 2, win 513, options [nop,nop,TS val 857742207 ecr 741124131], length 0 ^C 7 packets captured 7 packets received by filter 0 packets dropped by kernel root@ubuntu:~#
master节点
root@ubuntu:~# iptables -nvL -t nat | grep 30090 12 720 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nodeport-svc: */ tcp dpt:30090 12 720 KUBE-SVC-GFPAJ7EGCNM4QF4H tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nodeport-svc: */ tcp dpt:30090 root@ubuntu:~#
tartgetport
root@ubuntu:~# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 244d my-nginx ClusterIP 10.110.79.116 <none> 8280/TCP 37d my-nginx-np NodePort 10.99.1.231 <none> 8081:31199/TCP 36d nodeport-svc NodePort 10.97.11.232 <none> 3000:30090/TCP 60m web2 NodePort 10.110.171.213 <none> 8097:31866/TCP 20d
master 节点
root@ubuntu:~# telnet 10.97.11.232 3000
Trying 10.97.11.232...
Connected to 10.97.11.232.
Escape character is '^]'.
^CConnection closed by foreign host.
root@ubuntu:~#
root@ubuntu:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if673: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether ee:ee:d4:3a:73:67 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.0.22/24 brd 10.244.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::ecee:d4ff:fe3a:7367/64 scope link valid_lft forever preferred_lft forever root@ubuntu:~# tcpdump -i eth0 tcp and port 80 -ennvv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:38:01.400552 be:ca:86:22:86:b8 > ee:ee:d4:3a:73:67, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 47011, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.1.22003 > 10.244.0.22.80: Flags [S], cksum 0x162d (incorrect -> 0x6ed9), seq 2804599892, win 64240, options [mss 1460,sackOK,TS val 1489413550 ecr 0,nop,wscale 7], length 0 15:38:01.400599 ee:ee:d4:3a:73:67 > be:ca:86:22:86:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.22.80 > 10.244.0.1.22003: Flags [S.], cksum 0x162d (incorrect -> 0xba83), seq 2185197484, ack 2804599893, win 64308, options [mss 1410,sackOK,TS val 3737181061 ecr 1489413550,nop,wscale 7], length 0 15:38:01.400652 be:ca:86:22:86:b8 > ee:ee:d4:3a:73:67, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 47012, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.1.22003 > 10.244.0.22.80: Flags [.], cksum 0x1625 (incorrect -> 0xe25c), seq 1, ack 1, win 502, options [nop,nop,TS val 1489413550 ecr 3737181061], length 0 15:38:04.725599 be:ca:86:22:86:b8 > ee:ee:d4:3a:73:67, ethertype IPv4 (0x0800), length 71: (tos 0x10, ttl 64, id 47013, offset 0, flags [DF], proto TCP (6), length 57) 10.244.0.1.22003 > 10.244.0.22.80: Flags [P.], cksum 0x162a (incorrect -> 0xcf5f), seq 1:6, ack 1, win 502, options [nop,nop,TS val 1489416875 ecr 3737181061], length 5: HTTP 15:38:04.725633 ee:ee:d4:3a:73:67 > be:ca:86:22:86:b8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 3162, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.22.80 > 10.244.0.1.22003: Flags [.], cksum 0x1625 (incorrect -> 0xc85c), seq 1, ack 6, win 503, options [nop,nop,TS val 3737184386 ecr 1489416875], length 0 15:38:04.725812 ee:ee:d4:3a:73:67 > be:ca:86:22:86:b8, ethertype IPv4 (0x0800), length 375: (tos 0x0, ttl 64, id 3163, offset 0, flags [DF], proto TCP (6), length 361) 10.244.0.22.80 > 10.244.0.1.22003: Flags [P.], cksum 0x175a (incorrect -> 0x1b1e), seq 1:310, ack 6, win 503, options [nop,nop,TS val 3737184386 ecr 1489416875], length 309: HTTP, length: 309 HTTP/1.1 400 Bad Request Server: nginx/1.21.0 Date: Fri, 18 Jun 2021 07:38:04 GMT Content-Type: text/html Content-Length: 157 Connection: close <html> <head><title>400 Bad Request</title></head> <body> <center><h1>400 Bad Request</h1></center> <hr><center>nginx/1.21.0</center> </body> </html> 15:38:04.725918 be:ca:86:22:86:b8 > ee:ee:d4:3a:73:67, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 47014, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.1.22003 > 10.244.0.22.80: Flags [.], cksum 0x1625 (incorrect -> 0xc729), seq 6, ack 310, win 501, options [nop,nop,TS val 1489416875 ecr 3737184386], length 0 15:38:04.725949 ee:ee:d4:3a:73:67 > be:ca:86:22:86:b8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 3164, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.22.80 > 10.244.0.1.22003: Flags [F.], cksum 0x1625 (incorrect -> 0xc726), seq 310, ack 6, win 503, options [nop,nop,TS val 3737184386 ecr 1489416875], length 0 15:38:04.726018 be:ca:86:22:86:b8 > ee:ee:d4:3a:73:67, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 47015, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.1.22003 > 10.244.0.22.80: Flags [F.], cksum 0x1625 (incorrect -> 0xc727), seq 6, ack 311, win 501, options [nop,nop,TS val 1489416875 ecr 3737184386], length 0 15:38:04.726040 ee:ee:d4:3a:73:67 > be:ca:86:22:86:b8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 3165, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.22.80 > 10.244.0.1.22003: Flags [.], cksum 0x1625 (incorrect -> 0xc725), seq 311, ack 7, win 503, options [nop,nop,TS val 3737184386 ecr 1489416875], length 0
第二次telnet 转发到worker节点
root@ubuntu:~# telnet 10.97.11.232 3000 Trying 10.97.11.232... Connected to 10.97.11.232. Escape character is '^]'. ^CConnection closed by foreign host. root@ubuntu:~#
root@cloud:~# tcpdump -i eth0 tcp and port 80 -ennvv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:38:39.005965 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 63, id 54603, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.0.19739 > 10.244.2.6.80: Flags [S], cksum 0xa6c7 (correct), seq 1688161016, win 64240, options [mss 1460,sackOK,TS val 1489451152 ecr 0,nop,wscale 7], length 0 15:38:39.006028 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.2.6.80 > 10.244.0.0.19739: Flags [S.], cksum 0x181c (incorrect -> 0x95c3), seq 353214560, ack 1688161017, win 64308, options [mss 1410,sackOK,TS val 2686171991 ecr 1489451152,nop,wscale 7], length 0 15:38:39.006188 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 54604, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.19739 > 10.244.2.6.80: Flags [.], cksum 0xbd9c (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 1489451152 ecr 2686171991], length 0 15:38:41.193491 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 71: (tos 0x10, ttl 63, id 54605, offset 0, flags [DF], proto TCP (6), length 57) 10.244.0.0.19739 > 10.244.2.6.80: Flags [P.], cksum 0xaf11 (correct), seq 1:6, ack 1, win 502, options [nop,nop,TS val 1489453339 ecr 2686171991], length 5: HTTP 15:38:41.193534 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 63679, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.19739: Flags [.], cksum 0x1814 (incorrect -> 0xac7f), seq 1, ack 6, win 503, options [nop,nop,TS val 2686174179 ecr 1489453339], length 0 15:38:41.193606 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 375: (tos 0x0, ttl 64, id 63680, offset 0, flags [DF], proto TCP (6), length 361) 10.244.2.6.80 > 10.244.0.0.19739: Flags [P.], cksum 0x1949 (incorrect -> 0x0423), seq 1:310, ack 6, win 503, options [nop,nop,TS val 2686174179 ecr 1489453339], length 309: HTTP, length: 309 HTTP/1.1 400 Bad Request Server: nginx/1.19.5 Date: Fri, 18 Jun 2021 07:38:41 GMT Content-Type: text/html Content-Length: 157 Connection: close <html> <head><title>400 Bad Request</title></head> <body> <center><h1>400 Bad Request</h1></center> <hr><center>nginx/1.19.5</center> </body> </html> 15:38:41.193659 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 63681, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.19739: Flags [F.], cksum 0x1814 (incorrect -> 0xab49), seq 310, ack 6, win 503, options [nop,nop,TS val 2686174179 ecr 1489453339], length 0 15:38:41.193725 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 54606, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.19739 > 10.244.2.6.80: Flags [.], cksum 0xab4c (correct), seq 6, ack 310, win 501, options [nop,nop,TS val 1489453339 ecr 2686174179], length 0 15:38:41.193799 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 54607, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.19739 > 10.244.2.6.80: Flags [F.], cksum 0xab4a (correct), seq 6, ack 311, win 501, options [nop,nop,TS val 1489453339 ecr 2686174179], length 0 15:38:41.193818 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 63682, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.19739: Flags [.], cksum 0x1814 (incorrect -> 0xab48), seq 311, ack 7, win 503, options [nop,nop,TS val 2686174179 ecr 1489453339], length 0 ^C 10 packets captured 10 packets received by filter 0 packets dropped by kernel root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 82:ea:86:37:c3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.2.6/24 brd 10.244.2.255 scope global eth0 valid_lft forever preferred_lft forever root@cloud:~#
root@ubuntu:~# conntrack -L -o ktimestamp | grep 19739 tcp 6 26 TIME_WAIT src=10.10.16.82 dst=10.97.11.232 sport=46522 dport=3000 src=10.244.2.6 dst=10.244.0.0 sport=80 dport=19739 [ASSURED] mark=0 use=1 conntrack v1.4.4 (conntrack-tools): 160 flow entries have been shown. root@ubuntu:~#
root@ubuntu:~# iptables -nvL -t nat | grep 10.97.11.232 3 180 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.97.11.232 /* default/nodeport-svc: cluster IP */ tcp dpt:3000 3 180 KUBE-SVC-GFPAJ7EGCNM4QF4H tcp -- * * 0.0.0.0/0 10.97.11.232 /* default/nodeport-svc: cluster IP */ tcp dpt:3000 root@ubuntu:~#
tcpdump vxlan
root@ubuntu:~# telnet 10.97.11.232 3000 Trying 10.97.11.232... Connected to 10.97.11.232. Escape character is '^]'. ^CConnection closed by foreign host. root@ubuntu:~# conntrack -L -o ktimestamp | grep 1222 tcp 6 76 TIME_WAIT src=10.10.16.82 dst=10.97.11.232 sport=48714 dport=3000 src=10.244.2.6 dst=10.244.0.0 sport=80 dport=1222 [ASSURED] mark=0 use=1 conntrack v1.4.4 (conntrack-tools): 160 flow entries have been shown. root@ubuntu:~#
root@cloud:~# conntrack -L -o ktimestamp | grep 1222 conntrack v1.4.4 (conntrack-tools): 7 flow entries have been shown. tcp 6 49 TIME_WAIT src=10.244.0.0 dst=10.244.2.6 sport=1222 dport=80 src=10.244.2.6 dst=10.244.0.0 sport=80 dport=1222 [ASSURED] mark=0 use=1 root@cloud:~#
root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 82:ea:86:37:c3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.2.6/24 brd 10.244.2.255 scope global eth0 valid_lft forever preferred_lft forever root@cloud:~# tcpdump -i eth0 tcp and port 80 -ennvv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:45:04.478443 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 63, id 63071, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.0.1222 > 10.244.2.6.80: Flags [S], cksum 0x2957 (correct), seq 2769809023, win 64240, options [mss 1460,sackOK,TS val 1489836624 ecr 0,nop,wscale 7], length 0 15:45:04.478509 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.2.6.80 > 10.244.0.0.1222: Flags [S.], cksum 0x181c (incorrect -> 0xa563), seq 2111359165, ack 2769809024, win 64308, options [mss 1410,sackOK,TS val 2686557464 ecr 1489836624,nop,wscale 7], length 0 15:45:04.478825 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 63072, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.1222 > 10.244.2.6.80: Flags [.], cksum 0xcd3c (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 1489836624 ecr 2686557464], length 0 15:45:05.380852 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 71: (tos 0x10, ttl 63, id 63073, offset 0, flags [DF], proto TCP (6), length 57) 10.244.0.0.1222 > 10.244.2.6.80: Flags [P.], cksum 0xc3b6 (correct), seq 1:6, ack 1, win 502, options [nop,nop,TS val 1489837526 ecr 2686557464], length 5: HTTP 15:45:05.380874 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 25132, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.1222: Flags [.], cksum 0x1814 (incorrect -> 0xc62a), seq 1, ack 6, win 503, options [nop,nop,TS val 2686558366 ecr 1489837526], length 0 15:45:05.380935 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 375: (tos 0x0, ttl 64, id 25133, offset 0, flags [DF], proto TCP (6), length 361) 10.244.2.6.80 > 10.244.0.0.1222: Flags [P.], cksum 0x1949 (incorrect -> 0x18d5), seq 1:310, ack 6, win 503, options [nop,nop,TS val 2686558366 ecr 1489837526], length 309: HTTP, length: 309 HTTP/1.1 400 Bad Request Server: nginx/1.19.5 Date: Fri, 18 Jun 2021 07:45:05 GMT Content-Type: text/html Content-Length: 157 Connection: close <html> <head><title>400 Bad Request</title></head> <body> <center><h1>400 Bad Request</h1></center> <hr><center>nginx/1.19.5</center> </body> </html> 15:45:05.380997 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 25134, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.1222: Flags [F.], cksum 0x1814 (incorrect -> 0xc4f4), seq 310, ack 6, win 503, options [nop,nop,TS val 2686558366 ecr 1489837526], length 0 15:45:05.381085 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 63074, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.1222 > 10.244.2.6.80: Flags [.], cksum 0xc4f7 (correct), seq 6, ack 310, win 501, options [nop,nop,TS val 1489837526 ecr 2686558366], length 0 15:45:05.381259 0a:f9:a2:7f:2f:2a > 82:ea:86:37:c3:8d, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 63, id 63075, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.1222 > 10.244.2.6.80: Flags [F.], cksum 0xc4f5 (correct), seq 6, ack 311, win 501, options [nop,nop,TS val 1489837526 ecr 2686558366], length 0 15:45:05.381277 82:ea:86:37:c3:8d > 0a:f9:a2:7f:2f:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 25135, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.1222: Flags [.], cksum 0x1814 (incorrect -> 0xc4f2), seq 311, ack 7, win 503, options [nop,nop,TS val 2686558367 ecr 1489837526], length 0
vxlan
root@cloud:~# tcpdump -i enahisic2i0 udp and host 10.10.16.82 -eennv tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:45:03.458416 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 64, id 48686, offset 0, flags [none], proto UDP (17), length 110) 10.10.16.82.34152 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 63070, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.0.1222 > 10.244.2.6.80: Flags [S], cksum 0x2d53 (correct), seq 2769809023, win 64240, options [mss 1460,sackOK,TS val 1489835604 ecr 0,nop,wscale 7], length 0 15:45:04.478372 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 64, id 48882, offset 0, flags [none], proto UDP (17), length 110) 10.10.16.82.34588 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 63071, offset 0, flags [DF], proto TCP (6), length 60) 10.244.0.0.1222 > 10.244.2.6.80: Flags [S], cksum 0x2957 (correct), seq 2769809023, win 64240, options [mss 1460,sackOK,TS val 1489836624 ecr 0,nop,wscale 7], length 0 15:45:04.478570 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 64, id 65355, offset 0, flags [none], proto UDP (17), length 110) 10.10.16.47.58500 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.2.6.80 > 10.244.0.0.1222: Flags [S.], cksum 0xa563 (correct), seq 2111359165, ack 2769809024, win 64308, options [mss 1410,sackOK,TS val 2686557464 ecr 1489836624,nop,wscale 7], length 0 15:45:04.478801 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 48883, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.82.34588 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 63072, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.1222 > 10.244.2.6.80: Flags [.], cksum 0xcd3c (correct), ack 1, win 502, options [nop,nop,TS val 1489836624 ecr 2686557464], length 0 15:45:05.380826 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 121: (tos 0x0, ttl 64, id 49016, offset 0, flags [none], proto UDP (17), length 107) 10.10.16.82.34588 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 71: (tos 0x10, ttl 64, id 63073, offset 0, flags [DF], proto TCP (6), length 57) 10.244.0.0.1222 > 10.244.2.6.80: Flags [P.], cksum 0xc3b6 (correct), seq 1:6, ack 1, win 502, options [nop,nop,TS val 1489837526 ecr 2686557464], length 5: HTTP 15:45:05.380894 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 65476, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.47.58500 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 25132, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.1222: Flags [.], cksum 0xc62a (correct), ack 6, win 503, options [nop,nop,TS val 2686558366 ecr 1489837526], length 0 15:45:05.380974 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 425: (tos 0x0, ttl 64, id 65477, offset 0, flags [none], proto UDP (17), length 411) 10.10.16.47.58500 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 375: (tos 0x0, ttl 63, id 25133, offset 0, flags [DF], proto TCP (6), length 361) 10.244.2.6.80 > 10.244.0.0.1222: Flags [P.], cksum 0x18d5 (correct), seq 1:310, ack 6, win 503, options [nop,nop,TS val 2686558366 ecr 1489837526], length 309: HTTP, length: 309 HTTP/1.1 400 Bad Request Server: nginx/1.19.5 Date: Fri, 18 Jun 2021 07:45:05 GMT Content-Type: text/html Content-Length: 157 Connection: close <html> <head><title>400 Bad Request</title></head> <body> <center><h1>400 Bad Request</h1></center> <hr><center>nginx/1.19.5</center> </body> </html> 15:45:05.381012 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 65478, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.47.58500 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 25134, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.1222: Flags [F.], cksum 0xc4f4 (correct), seq 310, ack 6, win 503, options [nop,nop,TS val 2686558366 ecr 1489837526], length 0 15:45:05.381071 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 49017, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.82.34588 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 63074, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.1222 > 10.244.2.6.80: Flags [.], cksum 0xc4f7 (correct), ack 310, win 501, options [nop,nop,TS val 1489837526 ecr 2686558366], length 0 15:45:05.381240 48:57:02:64:e7:ab > 9c:52:f8:67:c4:d3, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 49018, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.82.34588 > 10.10.16.47.8472: OTV, flags [I] (0x08), overlay 0, instance 1 3a:2b:ed:85:2f:74 > 72:d3:9a:47:fd:43, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 63075, offset 0, flags [DF], proto TCP (6), length 52) 10.244.0.0.1222 > 10.244.2.6.80: Flags [F.], cksum 0xc4f5 (correct), seq 6, ack 311, win 501, options [nop,nop,TS val 1489837526 ecr 2686558366], length 0 15:45:05.381298 9c:52:f8:67:c4:d3 > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 116: (tos 0x0, ttl 64, id 65479, offset 0, flags [none], proto UDP (17), length 102) 10.10.16.47.58500 > 10.10.16.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1 72:d3:9a:47:fd:43 > 3a:2b:ed:85:2f:74, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 25135, offset 0, flags [DF], proto TCP (6), length 52) 10.244.2.6.80 > 10.244.0.0.1222: Flags [.], cksum 0xc4f2 (correct), ack 7, win 503, options [nop,nop,TS val 2686558367 ecr 1489837526], length 0
worker节点也能访问service ip
root@cloud:~# telnet 10.97.11.232 3000 Trying 10.97.11.232... Connected to 10.97.11.232. Escape character is '^]'. ^CConnection closed by foreign host. root@cloud:~#
root@cloud:~# iptables -nvL -t nat | grep 10.97.11.232 1 60 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.97.11.232 /* default/nodeport-svc: cluster IP */ tcp dpt:3000 1 60 KUBE-SVC-GFPAJ7EGCNM4QF4H tcp -- * * 0.0.0.0/0 10.97.11.232 /* default/nodeport-svc: cluster IP */ tcp dpt:3000 root@cloud:~#