k8s 二进制部署启动apiserver 报错 Error: unknown flag: etcdservers ,启动 kubelet Kubelet: cannot create certificate signing request

 systemctl status kube-apiserver  启动失败

查看错误日志

cat /var/log/messages|grep kube-apiserver|grep -i error

Jan 11 11:22:44 m1 kube-apiserver: --logtostderr                      log to standard error instead of files
Jan 11 11:25:16 m1 kube-apiserver: Error: unknown flag: --etcdservers
Jan 11 11:25:16 m1 kube-apiserver: --alsologtostderr                  log to standard error as well as files
Jan 11 11:25:16 m1 kube-apiserver: --logtostderr    

【Error: unknown flag: --etcdservers】 说明我的字符串写错了；

我复制教材的pdf内容； --etcdservers 复制出来粘贴到记事本,发现少了【-】符号；

chrome浏览器复制pdf -换行 内容到notepad++ 少了【-】符号

所以注意复制内容的差异；

 

修正后启动成功

 https://www.jianshu.com/p/19e2d7cc94d6

作者辛苦了；文章步骤很清晰，但是有一句错误，会引发kubelet在master上启动不了；

启动kubelet报错误：【kubelet  failed to run Kubelet: cannot create certificate signing request】

CA证书和token对于kubelet启动的重要性，注意这三步正确就能够启动kubelet成功；
1、token.csv
cat > /opt/kubernetes/cfg/token.csv << EOF
b1dc586d69159ff4e3ef7efa9db60e48,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

2、创建用户
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
3、生成bootstrap.kubeconfig文件 （ca.pem填充不上，手动修改）
==========================
##设置环境变量
KUBE_APISERVER="https://172.16.210.53:6443" # apiserver IP:PORT
TOKEN="b1dc586d69159ff4e3ef7efa9db60e48" # 与token.csv里保持一致

# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

======================
上边的命令：-certificate-authority在bootstrap.kubeconfig文件中会变成certificate-authority-data: AFSADFADFADFADFD（一堆字符码）
###
可以直接vi编辑（如下模板可用），就没有certificate-authority-data字符码的问题；
#######################
[root@localhost bin]# cat bootstrap.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    server: https://192.168.79.140:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubelet-bootstrap
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
  user:
    token: b1dc586d69159ff4e3ef7efa9db60e48
###########################

============================

执行如下命令生成kube-proxy.kubeconfig文件， 
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

报错如下；原因是*.pem文件没有设置成功；授权失败；解决方法参考【二进制部署k8s完整步骤和修正步骤】

[root@localhost logs]# more kube-proxy.FATAL
Log file created at: 2022/01/15 18:08:18
Running on machine: localhost
Binary: Built with gc go1.13.9 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0115 18:08:18.940662 12704 server.go:497] error loading config file "/opt/kubernetes/cfg/kube-proxy.kubeconfig": v1.Config.AuthInfos: []v1.NamedAuthInfo: v1.Nam
edAuthInfo.AuthInfo: v1.AuthInfo.ClientKeyData: ClientCertificateData: decode base64: illegal base64 data at input byte 24, error found in #10 byte of ...|proxy.pe
m","client-k|..., bigger context ...|ificate-data":"/opt/kubernetes/ssl/kube-proxy.pem","client-key-data":"/opt/kubernetes/ssl/kube-proxy|

https://blog.csdn.net/weixin_39608791/article/details/108881130

https://blog.csdn.net/IvyXYW/article/details/115710665

【failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is fo
rbidden: User "10001" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
】

cat > /opt/kubernetes/cfg/token.csv << EOF
b1dc586d69159ff4e3ef7efa9db60e48,10001,"system:node-bootstrapper"   ## 错误在与没有指定用户
EOF
改为：
cat > /opt/kubernetes/cfg/token.csv << EOF
b1dc586d69159ff4e3ef7efa9db60e48,kubelet-bootstrap,10001,"system:nodebootstrapper"
EOF


重新生成授权[kubelet-bootstrap]
【kubectl delete clusterrolebindings kubelet-bootstrap】 删除旧的；

【kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap】

忽然断电后，etcd启动失败：修复
https://blog.csdn.net/u013958257/article/details/106978416

备份 cp -r  /var/lib/etcd/default.etcd /var/lib/etcd/default.etcd.bak
删除 rm -rf /var/lib/etcd/default.etcd/*
kubectl describe pod web-65b7447c7-cxnqn
 kubectl get pods -n kube-system
  systemctl status flanneld
 kubectl apply -f kube-flannel.yml
 kubectl get pods -n kube-system
 kubectl get nodes
 kubectl get pods
 kubectl describe pod web-65b7447c7-cxnqn

　　

[root@master ~]# kubectl logs pi-nb5ds
Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log pi-nb5ds)
[root@master ~]# kubectl create clusterrolebinding kubernetes --clusterrole=cluster-admin --user=kubernetes
clusterrolebinding.rbac.authorization.k8s.io/kubernetes created
[root@master ~]# kubectl logs pi-nb5ds
3.1415926535897932384626433832795028841971693993

　　

二进制部署k8s完整步骤和修正步骤
https://www.jianshu.com/p/19e2d7cc94d6

3. 启用 TLS Bootstrapping 机制
   创建上述配置文件中token文件：
  修正为：
cat > /opt/kubernetes/cfg/token.csv << EOF
b1dc586d69159ff4e3ef7efa9db60e48,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

五.3. 生成bootstrap.kubeconfig文件
不用执行kubectl命令，命令执行不会自动填充ca.pem内容；
改为直接使用如内容；

[root@localhost bin]# cat bootstrap.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    server: https://192.168.79.140:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubelet-bootstrap
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
  user:
    token: b1dc586d69159ff4e3ef7efa9db60e48


5.4.3  生成kube-proxy.kubeconfig文件  
不执行kubectl命令，因为会导致.pem不能自动填写；
改为
[root@localhost k8s]# cat kube-proxy.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    server: https://192.168.79.140:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kube-proxy
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kube-proxy
  user:
    client-certificate: /opt/kubernetes/ssl/kube-proxy.pem
    client-key: /opt/kubernetes/ssl/kube-proxy-key.pem

部署好部署CNI网络后，kube-flannel.yml；部署nginx测试一下

[root@localhost k8s]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@localhost k8s]# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
[root@localhost k8s]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-f89759699-fbdw9 1/1 Running 0 40s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 3h54m
service/nginx NodePort 10.0.0.4 <none> 80:31254/TCP 13s
浏览器：
http://192.168.79.140:31254/

　5.5部署CNI网络

关于cni和flanel安装规划
cni安装到每台node节点
flannel安装到master节点
https://www.cnblogs.com/TSir/p/12240825.html

　在master上执行kubectl get nodes查看结果是否READY

5.5部署CNI网络完成后即为完成；