1、kube-apiserver、kube-controller-manager启动文件里面的公钥文件、私钥文件竟然是一样的，还怎么配对，这是源文档的大坑 查看源文档kube-apiserver启动文件中公钥 cat etc/systemd/system/kube-apiserver.service --service-account-key-file=/etc/kubernetes/cert/ca-key.pem 查看源文档kube-controller-manager启动文件中私钥 cat /etc/systemd/system/kube-controller-manager.service --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem 发现公钥和私钥都是用的一个私钥，这是不可能配对的，而且原文中已经说明需要配对，说明如下： 原版文件在apiserver启动文字中已经说明 > --service-account-key-file：签名 ServiceAccount Token 的公钥文件，kube-controller-manager 的 --service-account-private-key-file 指定私钥文件，两者配对使用； 但是原版中apiserver、controller-manager这两个启动文字里面用的私有、公钥都是一个，所以怀疑出错， 按照另一篇对照修改的：https://www.cnblogs.com/effortsing/p/10312081.html，需要修改如下： 生成 service account key cd /etc/kubernetes/ openssl genrsa -out /etc/kubernetes/sa.key 2048 openssl rsa -in /etc/kubernetes/cert/sa.key -pubout -out /etc/kubernetes/cert/sa.pub ls /etc/kubernetes/pki/sa.* cd $HOME 分发service account key到所有master节点 subprocess.call(["ansible k8s -m copy -a 'src=/etc/kubernetes/sa.key dest=/etc/kubernetes/cert/ force=yes'"], shell=True) subprocess.call(["ansible k8s -m copy -a 'src=/etc/kubernetes/sa.pub dest=/etc/kubernetes/cert/ force=yes'"], shell=True) 修改kube-apiserver启动文件中公钥为sa.pub cat etc/systemd/system/kube-apiserver.service --service-account-private-key-file=/etc/kubernetes/cert/sa.pub 修改kube-controller-manager启动文件中私钥为sa.key cat /etc/systemd/system/kube-controller-manager.service --service-account-private-key-file=/etc/kubernetes/cert/sa.key 2、源文档kube-controller-manager启动文件里面少两个致命参数，导致flannel启动失败，缺少 --allocate-node-cidrs=true --cluster-cidr=172.30.0.0/16 flannel启动失败，报错如下 Error registering network: failed to acquire lease: node "test4" pod cidr not assigned 查看pod [root@test4 profile]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE kube-flannel-ds-gzvrh 0/1 Error 0 <invalid> 通过docker查看flannel日志 [root@test4 profile]# docker ps -l CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f7be3ebe77fd b949a39093d6 "/opt/bin/flanneld -…" 1 second ago Created k8s_kube-flannel_kube-flannel-ds-7cqww_kube-system_26fab004-2b88-11e9-9085-000c2935f634_0 [root@test4 profile]# docker logs f7be3ebe77fd I0208 09:58:34.068723 1 main.go:488] Using interface with name ens33 and address 192.168.0.94 I0208 09:58:34.069094 1 main.go:505] Defaulting external address to interface address (192.168.0.94) I0208 09:58:34.376952 1 kube.go:131] Waiting 10m0s for node controller to sync I0208 09:58:34.466001 1 kube.go:294] Starting kube subnet manager I0208 09:58:35.481478 1 kube.go:138] Node controller sync successful I0208 09:58:35.481666 1 main.go:235] Created subnet manager: Kubernetes Subnet Manager - test4 I0208 09:58:35.481694 1 main.go:238] Installing signal handlers I0208 09:58:35.482001 1 main.go:353] Found network config - Backend type: vxlan I0208 09:58:35.482255 1 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false E0208 09:58:35.483159 1 main.go:280] Error registering network: failed to acquire lease: node "test4" pod cidr not assigned I0208 09:58:35.483433 1 main.go:333] Stopping shutdownHandler... 看到：Error registering network: failed to acquire lease: node "test4" pod cidr not assigned 原因：是因为按照二进制高可用文档安装配置kube-controller-manager启动文件的时候，有坑，当时没有发现， 解决： 启动文件加上下面两句话，那篇文档没有加，所以报错；下面这个cluster-cidr要和kube-flannel.yml里面的地址一致，要和kube-proxy.config.yaml里面的clusterCIDR一致 --allocate-node-cidrs=true --cluster-cidr=172.30.0.0/16 4、源文档中的kubelet启动参数中都没有带cadvisor监控服务参数，源文档中就可以访问cadvisor监控了。明摆着安装完把cadvisor参数去掉了，这是大坑 5、执行查看资源报错： unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy) [root@test4 ~]# kubectl exec -it http-test-dm2-6dbd76c7dd-cv9qf sh error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy) 解决：创建apiserver到kubelet的权限 注意：user=kubernetes ，这个user要替换掉下面yaml文件里面的用户名 cat > apiserver-to-kubelet.yaml <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kubernetes-to-kubelet rules: - apiGroups: - "" resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:kubernetes namespace: "" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kubernetes-to-kubelet subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kubernetes EOF 创建授权: kubectl create -f apiserver-to-kubelet.yaml [root@test4 ~]# kubectl create -f apiserver-to-kubelet.yaml clusterrole.rbac.authorization.k8s.io/system:kubernetes-to-kubelet created clusterrolebinding.rbac.authorization.k8s.io/system:kubernetes created 重新进到容器查看资源 [root@test4 ~]# kubectl exec -it http-test-dm2-6dbd76c7dd-cv9qf sh / # exit 现在可以进到容器里面查看资源了 参照文档：https://www.jianshu.com/p/b3d8e8b8fd7e 6、源文档中kube-apiserver启动参数中没有这句话 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname 这句话是必须要加的否则用kubectl查看资源、创建dnstools工具时候会报如下错误： [root@test4 profile]# kubectl run -it --rm --image=infoblox/dnstools dns-client kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. If you don't see a command prompt, try pressing enter. Error attaching, falling back to logs: error dialing backend: dial tcp 0.0.0.0:10250: connect: connection refused deployment.apps "dns-client" deleted Error from server: Get https://test4:10250/containerLogs/default/dns-client-86c6d59f7-tzh5c/dns-client: dial tcp 0.0.0.0:10250: connect: connection refused [root@test4 ~]# kubectl exec -it http-test-dm2-6dbd76c7dd-cv9qf sh error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy) 7、源文档中kube-apiserver启动参数中--enable-admission-plugins= 这个选项里面的参数很少，导致各种错误，必须填写全，如下： --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota